Apparatus and methods for use with skeletal procedures

ABSTRACT

3D image data of a skeletal portion is acquired. A location of a proximal portion of a tool is calculated and a location is derived of a distal portion of the tool with respect to the skeletal portion, with respect to the 3D image data. A display indicates the derived location. First and second 2D images of the distal portion of the tool are acquired from two different poses of a 2D imaging device with respect to the subject and registered with the 3D image data. The location of the distal portion with respect to the 3D image data of the skeletal portion is determined based on the registration and an identified location of the distal portion within the 2D x-rays. Based upon the determined location, the display updates the indicated location of the distal portion. Other embodiments are also described.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority from:

U.S. Provisional Patent Application No. 62/307,514 to Tolkowsky, filedMar. 13, 2016, entitled “Freehand Assistant for Spinal Surgery;”

U.S. Provisional Patent Application No. 62/334,463 to Tolkowsky, filedMay 11, 2016, entitled “Freehand Assistant for Spinal Surgery;”

U.S. Provisional Patent Application No. 62/362,607 to Tolkowsky, filedJul. 15, 2016, entitled “Freehand Assistant for Spinal Surgery;”

U.S. Provisional Patent Application No. 62/398,085 to Tolkowsky, filedSep. 22, 2016, entitled “Freehand Assistant for Spinal Surgery;”

U.S. Provisional Patent Application No. 62/439,495 to Tolkowsky, filedDec. 28, 2016, entitled “Freehand Assistant for Spinal Surgery;” and

U.S. Provisional Patent Application No. 62/463,747 to Tolkowsky, filedFeb. 27, 2017, entitled “Freehand Assistant for Spinal Surgery.”

The above-referenced applications are incorporated herein by reference.

FIELD OF EMBODIMENTS OF THE INVENTION

Some applications of the present invention generally relate to medicalapparatus and methods. Specifically, some applications of the presentinvention relate to apparatus and methods for use in procedures that areperformed on skeletal anatomy.

BACKGROUND

Approximately 5 million spine surgeries are performed annuallyworldwide. Traditional, manual surgery is known as freehand surgery.Typically, for such procedures, a 3D scan (e.g., a CT and/or MRI) scanis performed prior to surgery. A CT scan is typically performed for bonytissue (e.g., vertebra), and an MRI scan is typically performed for softtissue (e.g., discs).

Reference is made to FIG. 1A, which is a schematic illustration of atypical set up of an orthopedic operating room, for procedures that areperformed in a freehand manner. Typically, in freehand procedures,although the CT and/or MRI scan is examined by the surgeon whenpreparing for surgery, no use is made of the CT and/or MRI images duringsurgery. Rather, the surgery is typically performed under 2D x-ray imageguidance (also referred to as fluoroscopic guidance), the 2D x-raystypically being acquired using an x-ray C-arm. FIG. 1A shows a surgeon10 performing a procedure using intraprocedural x-ray images that areacquired by a C-arm 34, and displayed on a display 12. Freehand surgeryin which there is significant use of x-rays is known asfluoroscopy-guided surgery. X-ray C-arms are ubiquitous, familiar tosurgeons, useful for acquiring real-time images, tool-neutral (i.e.,there is no requirement to use orthopedic tools that are adaptedspecifically for imaging by the x-ray C-arm), and relativelyinexpensive. A growing proportion of spinal surgeries are performedusing a minimally-invasive surgery (also known as “MIS,” or in the caseof spine surgery, minimally-invasive spine surgery, which is also knownas “MISS”), or “mini-open” surgery. In contrast to open surgery, inwhich an incision is made along the applicable segment of the spine uponwhich surgery is performed, in minimally-invasive surgery, very smallincisions are made at the insertion point of tools. In “mini-open”surgery, incisions are made that are smaller than in open surgery andlarger than in minimally-invasive surgery. Typically, the less invasivethe type of surgery that is performed, the greater the use of x-rayimaging for assisting the procedure. There is evidence that lessinvasive procedures that are performed under fluoroscopic guidance aremore accurate than open procedures. However, the use of real-timefluoroscopic guidance typically exposes the patient, as well as thesurgeon and the support staff to a relatively large amount of harmfulradiation.

A minority of procedures are performed using Computer Aided Surgery(CAS) systems that provide navigation and/or robotics. Such systemstypically make use of CT and/or MRI images that are generated before thepatient is in the operating room, or when the patient is within theoperating room, but typically before an intervention has commenced. TheCT and/or MRI images are registered to the patient's body, and, duringsurgery, tools are navigated upon the images, the tools being movedmanually, robotically or both.

Typically, in CAS procedures, a uniquely-identifiable location sensor isattached to each tool that needs to be tracked by the CAS system. Eachtool is identified and calibrated at the beginning of the procedure. Inaddition, a uniquely-identifiable reference sensor is rigidly attachedto the organ. In the case of spinal surgery, the reference sensor istypically drilled into the sacrum or spine, and, if surgery is performedalong a number of vertebrae, the reference sensor is sometimes moved anddrilled into a different portion of the spine, mid-surgery, in order toalways be close to the surgical site. The images to be navigated upon(e.g., CT, MRI), which are acquired before the patient is in theoperating room, or when the patient is within the operating room, butbefore an intervention has commenced, are registered to the patient'sbody or a portion thereof. In order to register the images to thepatient's body, the current location of the patient's body is broughtinto the same reference frame of coordinates as the images using thereference sensor. The location sensors on the tools and the referencesensor on the patient's body are then tracked in order to determine thelocations of the tools relative to the patient's body, and a symbolicrepresentation of the tool is displayed upon the images that arenavigated upon. Typically, the tool is tracked in 5-6 degrees offreedom.

There are various techniques that are utilized for the tracking oftools, and corresponding location sensors are used for each technique.One technique is infrared (“IR”) tracking, whereby an array of camerastrack active IR lights on the tools and the patient's body, or an arrayof beams and cameras tracks passive IR reflectors on the tools and thepatient's body. In both categories of IR tracking, lines of sight mustbe maintained at all times between the tracker and the tools. Forexample, if the line of sight is blocked by the surgeon's hands, thiscan interfere with the tracking. Another technique is electromagnetic ormagnetic tracking, whereby a field generator tracks receivers, typicallycoils, on the tools and the patient's body. For those latter techniques,environmental interferences from other equipment much be avoided. Ineach of the techniques, the location sensors of the navigation systemare tracked using tracking components that would not be present in theoperating room in the absence of the navigation system (i.e., thelocation sensors do not simply rely upon imaging by imaging devices thatare typically used in an orthopedic operating room in the absence of thenavigation system).

A further technique that can be used with a robotically-driven tool isto start with the tool at a known starting point relative to thepatient's body, and to then record motion of the tool from the startingpoint. Alternatively, such tools can be tracked using theabove-described techniques.

Given the nature of CAS procedures, the equipment required for suchprocedures is typically more expensive than that of non-CAS procedures(non-CAS procedures including open procedures, mini-open procedures, orminimally-invasive procedures that are not computer aided with respectto the guidance of tools). Such procedures typically limit toolselection to those fitted with location sensors as described above, andtypically require such tools to be individually identified andcalibrated at the beginning of each surgery.

SUMMARY OF EMBODIMENTS

In accordance with some applications of the present invention, thefollowing steps are typically performed during procedures that areperformed on skeletal anatomy, using a system that includes a computerprocessor. Such procedures may include joint (e.g., shoulder, knee, hip,and/or ankle) replacement, joint repair, fracture repair (e.g., femur,tibia, and/or fibula), a procedure that is performed on a rib (e.g., ribremoval, or rib resection), and/or other interventions in which 3D imagedata are acquired prior to the intervention and 2D images are acquiredduring the intervention. For some applications, the steps are performedduring a procedure that is performed on one or more vertebrae of asubject's spine.

Typically, in a first step, targeted vertebra(e) are marked by anoperator with respect to 3D image data (e.g., a 3D image, a 2Dcross-section derived from 3D image data, and/or a 2D projection imagederived from 3D image data) of the subject's spine. For someapplications, in a second step, sets of markers are placed on thesubject, underneath the subject, on the surgical table, or above thesurgical table in a vicinity of the subject. Typically, in a third step,vertebrae of the spine are identified in order to verify that theprocedure is being performed with respect to the correct vertebra (astep which is known as “level verification”), using radiographic imagesof the spine and the markers to facilitate the identification. For someapplications, in a fourth step, an incision site (in the case ofminimally-invasive surgery), or a tool entry point into a vertebra (inthe case of open surgery) is determined. In a fifth step, the first toolin the sequence of tools (which is typically a needle, e.g., a Jamshidi™needle) is typically inserted into the subject (e.g., in the subject'sback) via the incision site or the tool entry point, and is slightlyfixated in the vertebra. In a sixth step, two or more 2D radiographicimages are typically acquired from respective views that typicallydiffer by at least 10 degrees (and further typically by 30 degrees ormore), and one of which is typically from the direction of insertion ofthe tool. Typically, generally-AP and generally-lateral images areacquired. Alternatively or additionally, images from different views areacquired. Typically, in a seventh step, the computer processor registersthe 3D image data to the 2D images.

Typically, 3D image data and 2D images of individual vertebrae areregistered to each other. Further typically, the 3D image data and 2Dimages are registered to each other by generating a plurality of 2Dprojections from the 3D image data, and identifying respective first andsecond 2D projections that match each of the 2D x-ray images of thevertebra, as described in further detail hereinbelow. Typically, firstand second 2D x-ray images of the vertebra are acquired using an x-rayimaging device that is unregistered with respect to the subject's body,by (a) acquiring a first 2D x-ray image of the vertebra (and at least aportion of the tool) from a first view, while the x-ray imaging deviceis disposed at a first pose with respect to the subject's body, (b)moving the x-ray imaging device to a second pose with respect to thesubject's body, and (c) while the x-ray imaging device is at the secondpose, acquiring a second 2D x-ray image of at least the portion of thetool and the vertebra from a second view. For some applications, morethan two 2D x-rays are acquired from respective x-ray image views, andthe 3D image data and 2D x-ray images are typically all registered toeach other by identifying a corresponding number of 2D projections ofthe 3D image data that match respective 2D x-ray images.

For some applications, the computer processor acquires a 2D x-ray imageof a tool inside the vertebra from only a single x-ray image view, andthe 2D x-ray image is registered to the 3D image data by generating aplurality of 2D projections from the 3D image data, and identifying a 2Dprojection that matches the 2D x-ray image of the vertebra. In responseto registering the 2D x-ray image to the 3D image data, the computerprocessor drives a display to display a cross-section derived from the3D image data at a current location of a tip of the tool, as identifiedfrom the 2D x-ray image, and optionally to show a vertical line on thecross-sectional image indicating a line within the cross-sectional imagesomewhere along which the tip of the tool is currently disposed.

As described hereinabove, typically two or more 2D x-rays are acquiredfrom respective x-ray image views, and the 3D image data and 2D imagesare typically registered to each other by identifying a correspondingnumber of 2D projections of the 3D image data that match respective 2Dx-ray images. Subsequent to the registration of the 3D image data to the2D x-ray images, additional features of the system are applied by thecomputer processor. For example, the computer processor may drive thedisplay to display the anticipated (i.e., extrapolated) path of the toolwith reference to a target location and/or with reference to a desiredinsertion vector. For some applications, the computer processorsimulates tool progress within a secondary 2D imaging view, based uponobserved progress of the tool in a primary 2D imaging view.Alternatively or additionally, the computer processor overlays an imageof the tool, a representation thereof, and/or a representation of thetool path, upon the 3D image data (e.g., a 3D image, a 2D cross-sectionderived from 3D image data, and/or a 2D projection image derived from 3Dimage data), the location of the tool or tool path having been derivedfrom current 2D images.

As described hereinabove, for some applications, sets of markers areplaced on the subject, underneath the subject, on the surgical table, orabove the surgical table. Typically, the markers that are placed atrespective locations with respect to the subject are identifiable inx-ray images, in optical images, and physically to the human eye. Forexample, respective radiopaque alphanumeric characters may be placed atrespective locations. For some applications, markers placed atrespective locations are identifiable based upon other features, e.g.,based upon the dispositions of the markers relative to other markers.Using a radiographic imaging device, a plurality of radiographic imagesof the set of radiopaque markers are acquired, respective images beingof respective locations along at least a portion of the subject's spineand each of the images including at least some of the radiopaquemarkers. Using the computer processor, locations of the radiopaquemarkers within the radiographic images are identified, by means of imageprocessing. At least some of the radiographic images are combined withrespect to one another based upon the identified locations of theradiopaque markers within the radiographic images. Typically, suchcombination of images is similar to stitching of images. However, theimages are typically not precisely stitched such as to stitch portionsof the subject's anatomy in adjacent images to one another. Rather, theimages are combined with sufficient accuracy to be able to determine alocation of the given vertebra within the combined radiographic images.The computer processor thus automatically determines (or facilitatesmanual determination of) a location of a given vertebra within thecombined radiographic images. Based upon the location of the givenvertebra within the combined radiographic images, a location of thegiven vertebra in relation to the set of radiopaque markers that isplaced on the subject is determined, as described in further detailhereinbelow. The markers are typically utilized to provide additionalfunctionalities, or in some cases to facilitate functionalities, asdescribed in further detail hereinbelow.

There is therefore provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theskeletal portion,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured, while a portion of the tool is disposedat a first location along the longitudinal insertion path, tosequentially:

-   -   acquire a first 2D x-ray image of at least the portion of the        tool and the skeletal portion from a first view, while the 2D        x-ray imaging device is disposed at a first pose with respect to        the subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least the portion of the tool and        the skeletal portion from a second view, and

(c) an output device,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of the skeletal portion from the 3D        imaging device,    -   receive the first and second 2D x-ray images of at least the        portion of the tool and the skeletal portion from the 2D x-ray        imaging device,    -   register the first and second 2D x-ray images to the 3D image        data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second 2D projections that            match the first and second 2D x-ray images of the skeletal            portion,    -   identify a location of the portion of the tool with respect to        the skeletal portion, within the first and second 2D x-ray        images, by means of image processing,    -   based upon the identified location of the portion of the tool        within the first and second 2D x-ray images, and the        registration of the first and second 2D x-ray images to the 3D        image data, determine the first location of the portion of the        tool with respect to the 3D image data,    -   subsequent to moving the portion of the tool to a second        location along the longitudinal insertion path with respect to        the skeletal portion, receive one or more additional 2D x-ray        images of at least the portion of the tool and the skeletal        portion from the 2D x-ray imaging device, the one or more        additional 2D x-ray images being acquired from a single image        view,    -   identify the second location of the portion of the tool within        the one or more additional 2D x-ray images, by means of image        processing,    -   derive the second location of the portion of the tool with        respect to the 3D image data, based upon the second location of        the portion of the tool within the one or more additional 2D        x-ray images, and the determined first location of the portion        of the tool with respect to the 3D image data, and    -   generate an output on the output device, at least partially in        response thereto.

In some applications, the at least one computer processor is configuredto receive the one or more additional 2D x-ray images of at least theportion of the tool and the skeletal portion from the 2D x-ray imagingdevice that are acquired from the single image view by receiving one ormore additional 2D x-ray images of at least the portion of the tool andthe skeletal portion from one of the first and second image views. Insome applications, the at least one computer processor is configured toreceive the one or more additional 2D x-ray images of at least theportion of the tool and the skeletal portion from the 2D x-ray imagingdevice that are acquired from the single image view by receiving one ormore additional 2D x-ray images of at least the portion of the tool andthe skeletal portion from a third image view that is different from thefirst and second image views.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured, while a portion of the tool is disposedat a first location along the longitudinal insertion path, tosequentially:

-   -   acquire a first 2D x-ray image of at least the portion of the        tool and the skeletal portion from a first view, while the 2D        x-ray imaging device is disposed at a first pose with respect to        the subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least the portion of the tool and        the skeletal portion from a second view, and

(b) an output device,

the apparatus including:

at least one computer processor configured:

-   -   to receive the first and second 2D x-ray images of at least the        portion of the tool and the skeletal portion from the 2D x-ray        imaging device,    -   to identify the portion of the tool in the first and second 2D        x-ray images, by means of image processing,    -   to determine a relationship between the first and second 2D        x-ray images,    -   subsequent to the tool being advanced along the longitudinal        insertion path with respect to the skeletal portion, such that        the portion of the tool is disposed at a second location along        the longitudinal insertion path, to receive one or more        additional 2D x-ray images of at least the portion of the tool        and the skeletal portion from the 2D x-ray imaging device, the        one or more additional 2D x-ray images being acquired from a        single image view,    -   to identify the second location of the portion of the tool        within the one or more additional 2D x-ray images, by means of        image processing,    -   to derive the second location of the portion of the tool with        respect to at least one of the first and second 2D x-ray images,        based upon the second location of the portion of the tool that        was identified within the one or more additional 2D x-ray        images, and the determined relationship between the first and        second 2D x-ray images, and    -   to generate an output on the output device, at least partially        in response thereto.

In some applications, the at least one computer processor is configuredto determine the relationship between the first and second 2D x-rayimages by registering the first and second 2D x-ray images to 3D imagedata of the skeletal portion. In some applications, the apparatus is foruse with a three-dimensional radiopaque jig, and the at least onecomputer processor is configured to determine the relationship betweenthe first and second 2D x-ray images using the three-dimensionalradiopaque jig that is visible in the first and second 2D x-ray images.In some applications, the tool includes two or more radiopaque featuresthat are visible in the first and second 2D x-ray images, and the atleast one computer processor is configured to determine the relationshipbetween the first and second 2D x-ray images using the two or moreradiopaque portions of the tool that are visible in the first and second2D x-ray images. In some applications, the at least one computerprocessor is configured to receive the one or more additional 2D x-rayimages of at least the portion of the tool and the skeletal portion fromthe 2D x-ray imaging device by receiving one or more additional 2D x-rayimages of at least the portion of the tool and the skeletal portion fromone of the first and second image views. In some applications, the atleast one computer processor is configured to receive the one or moreadditional 2D x-ray images of at least the portion of the tool and theskeletal portion from the 2D x-ray imaging device by receiving one ormore additional 2D x-ray images of at least the portion of the tool andthe skeletal portion from a third image view that is different from thefirst and second image views.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure with respect toa given vertebra of a spine of a subject, and for use with aradiographic imaging device, the apparatus including:

a set of radiopaque markers configured to be placed in a vicinity of thesubject, such that markers that are placed at respective locations withrespect to the subject are identifiable; and

at least one computer processor configured:

-   -   to receive, from the radiographic imaging device, a plurality of        radiographic images of the set of radiopaque markers, respective        images being of respective locations along at least a portion of        the subject's spine and each of the images including at least        some of the radiopaque markers,    -   to identify locations of the radiopaque markers within the        radiographic images, by means of image processing,    -   to combine at least some of the radiographic images with respect        to one another based upon the identified locations of the        radiopaque markers within the radiographic images,    -   to automatically determine a location of the given vertebra        within the combined radiographic images, by means of image        processing, and    -   to generate an output in response thereto.

In some applications, the apparatus is for use with a display, and theat least one computer processor is configured to generate the output bydriving the display to display the combined radiographic images with anindication of the given vertebra displayed with respect to the combinedradiographic images. In some applications, the apparatus is for use witha display, and the at least one computer processor is configured togenerate the output by driving the display to display the combinedradiographic images with an indication of the given vertebra displayedwith respect to the combined radiographic images, and driving thedisplay to display an indication of the given vertebra with respect to3D image data of at least a portion of the subject's spine.

In some applications, the at least one computer processor is configuredto automatically determine the location of the given vertebra within thecombined radiographic images by means of image processing, by:identifying an identifiable feature within the combined radiographicimage; identifying individual vertebra within the combined radiographicimage; and counting vertebra from the identifiable feature. In someapplications, the at least one computer processor is configured toidentify the identifiable feature within the combined radiographic imageby identifying a sacrum of the subject within the combined radiographicimage.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure with respect toa given rib of a ribcage of a subject, and for use with a radiographicimaging device, the apparatus including:

a set of radiopaque markers configured to be placed in a vicinity of thesubject, such that markers that are placed at respective locations withrespect to the subject are identifiable; and

at least one computer processor configured:

-   -   to receive, from the radiographic imaging device, a plurality of        radiographic images of the set of radiopaque markers, respective        images being of respective locations along at least a portion of        the subject's ribcage and each of the images including at least        some of the radiopaque markers,    -   to identify locations of the radiopaque markers within the        radiographic images, by means of image processing,    -   to combine at least some of the radiographic images with respect        to one another based upon the identified locations of the        radiopaque markers within the radiographic images    -   to automatically determine a location of the given rib within        the combined radiographic images, by means of image processing,        and    -   to generate an output in response thereto.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure on a spine of abody of a subject, the apparatus including:

a first set of radiopaque markers;

a second set of radiopaque markers; and

one or more surfaces, the first and second sets of radiopaque markersbeing coupled to one another via the one or more surfaces, the one ormore surfaces being configured:

-   -   to position the first and second sets of markers on respective        sides of the subject's spine at predefined positions with        respect to each other, by the one or more surfaces being placed        over a portion of the subject's spine upon which the procedure        is to be performed; and    -   subsequent to positioning the first and second set of markers on        the respective sides of the subject's spine, to be removable        from the subject's body, such as to facilitate performance of        the procedure upon the portion of the subject's spine over which        the one or more surfaces were placed, while leaving the first        and second sets of markers on the respective sides of the        subject's spine at the predefined position with respect to each        other.

In some applications, at least some of the markers of each of the firstand second sets of markers are rigid and have known dimensions. In someapplications, each of the first and second sets of markers includes aset of radiopaque characters, the sets being different from each other.In some applications, the apparatus further includes a third set of aradiopaque markers that is coupled to the first and second set ofmarkers via the one or more surfaces such that the third set of markersis configured to be positioned along a center of the subject's spine,when the first and second sets of markers are positioned on respectivesides of the subject's spine at the predefined positions with respect toeach other, by the one or more surfaces being placed over the portion ofthe subject's spine upon which the procedure is to be performed.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with a radiographic imaging devicethat is configured to acquire radiographic images of a spine of a bodyof a subject, the apparatus including:

a flexible material configured to be placed upon the subject's bodyalong at least a portion of the subject's spine, and to generallyconform to contours of the subject's body; and

a set of radiopaque markers disposed upon the flexible material, atleast some of the radiopaque markers being rigid and having respectiveknown dimensions, the rigid radiopaque markers being disposed upon theflexible material, such that rigidity of the rigid markers does notprevent the flexible material from generally conforming to the contoursof the subject's body; and

at least one computer processor configured to:

-   -   identify, by means of image processing, at least one of the        rigid radiopaque markers within a radiographic image of the        spine acquired by the radiographic imaging device, and    -   based upon the identified rigid radiopaque markers, determine        dimensions of features within the radiographic image, by means        of image processing.

There is further provided, in accordance with some applications of thepresent invention, a method for performing a procedure on a spine of abody of a subject, the method including:

providing:

-   -   a first set of radiopaque markers, and    -   a second set of radiopaque markers, and

positioning the first and sets of markers with respect to the subject'sspine, such that the first set of the markers appear in radiographicimages of the subject's spine that are acquired from a first image view,and such that the second set of the markers appear in radiographicimages of the subject's spine that are acquired from a second imageview;

acquiring radiographic images of the subject's spine from the first andsecond image views; and

associating a given vertebra that appears in the radiographic images ofthe spine from the first image view, with the given vertebra of thespine in the radiographic images of the spine from the second imageview, by identifying markers that have a known association with oneanother in the radiographic images acquired from the first and secondimage views.

In some applications, associating the given vertebra that appears in theradiographic images of the spine from the first image view, with thegiven vertebra of the spine in the radiographic images of the spine fromthe second image view includes, using at least one computer processor,associating the given vertebra that appears in the radiographic imagesof the spine from the first image view, with the given vertebra of thespine in the radiographic images of the spine from the second imageview, by identifying markers that have a known association with oneanother in the radiographic images acquired from the first and secondimage views. In some applications, associating the given vertebra thatappears in the radiographic images of the spine from the first imageview, with the given vertebra of the spine in the radiographic images ofthe spine from the second image view includes, manually, associating thegiven vertebra that appears in the radiographic images of the spine fromthe first image view, with the given vertebra of the spine in theradiographic images of the spine from the second image view, byidentifying markers that have a known association with one another inthe radiographic images acquired from the first and second image views

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure on a spine of abody of a subject, the apparatus including:

a first set of radiopaque markers configured to be placed with respectto the subject's spine such that the first set of radiopaque markersappear in radiographic images of the subject's spine that are acquiredfrom a first image view; and

a second set of radiopaque markers,

-   -   the first and second sets of markers being coupled to one        another such that when first set of radiopaque markers are        placed with respect to the subject's spine such that the first        set of radiopaque markers appear in radiographic images of the        subject's spine that are acquired from the first image view, the        second set of the markers appear in radiographic images of the        subject's spine that are acquired from a second image view,    -   the first and second sets of markers thereby facilitating        associating a given vertebra that appears in the radiographic        images of the spine from the first image view, with the given        vertebra of the spine in the radiographic images of the spine        from the second image view, by facilitating identification of        markers that have a known association with one another in the        radiographic images acquired from the first and second image        views.

In some applications, the apparatus further includes at least onecomputer processor that is configured to associate the given vertebrathat appears in the radiographic images of the spine from the firstimage view with the given vertebra of the spine in the radiographicimages of the spine from the second image view, by identifying markersthat have the known association with one another in the radiographicimages acquired from the first and second image views.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure with respect toa skeletal portion within a body of a subject, using a 2D radiographicimaging device, an optical camera, and one or more displays, theapparatus including:

a set of radiopaque markers configured to be placed in a vicinity of thesubject; and

at least one computer processor configured:

-   -   to receive a radiographic image of the skeletal portion from the        2D radiographic imaging device,    -   to receive an optical image of the subject's body from the        optical camera;    -   to identify the radiopaque markers in the radiographic image and        in the optical image, by means of image processing,    -   based upon the identification of the radiopaque markers in the        radiographic image and in the optical image, to bidirectionally        map the radiographic image and the optical image with respect to        one another,    -   to display the radiographic image and the optical image        separately from one another, upon the one or more displays,    -   to receive an input indicating a location in a first one of the        radiographic and the optical images, and    -   in response thereto, to generate an output indicating the        location in the other one of the radiographic and the optical        images.

In some applications, the at least one computer processor is configuredto receive the input indicating the location in the first one of theradiographic and the optical images by identifying, within the opticalimage of the subject's body, an object placed at a proposed entry pointinto the skeletal portion, and the at least one computer processor isconfigured to generate the output by generating an output indicating theproposed entry point into the skeletal portion with respect to theradiographic image. In some applications, the at least one computerprocessor is configured to receive the input indicating the location inthe first one of the radiographic and the optical images by identifying,within the optical image of the subject's body, an object placed at aproposed incision site, and the at least one computer processor isconfigured to generate the output by generating an output indicating theproposed incision site in the radiographic image.

In some applications, the at least one computer processor is configuredto receive the input indicating the location in the first one of theradiographic and the optical images by receiving an input indicating alocation in the optical image, the at least one computer processor beingfurther configured, in response to receiving the input, to drive the oneor more displays to display a cross-section of the skeletal portioncorresponding to the indicated location. In some applications, the atleast one computer processor is further configured, in response toreceiving the input, to drive the one or more displays to display a lineupon the cross-section of the skeletal portion, indicating that theindicated location is somewhere along the line.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theskeletal portion,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured, while a portion of the tool is disposedat a location along the longitudinal insertion path, to sequentially:

-   -   acquire a first 2D x-ray image of at least the portion of the        tool and the skeletal portion from a first view, while the 2D        x-ray imaging device is disposed at a first pose with respect to        the subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least the portion of the tool and        the skeletal portion from a second view, and

(c) an output device,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of the skeletal portion from the 3D        imaging device,    -   receive the first and second 2D x-ray images of at least the        portion of the tool and the skeletal portion from the 2D x-ray        imaging device,    -   register the first and second 2D x-ray images to the 3D image        data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second 2D projections that            match the first and second 2D x-ray images,    -   identify, the location of the portion of the tool with respect        to the skeletal portion, within the first and second x-ray        images, by means of image processing,    -   based upon the identified location of the portion of the tool        within the first and second x-ray images, and the registration        of the first and second 2D x-ray images to the 3D image data,        derive a relationship between the location of the portion of the        tool with respect to the 3D image data and a given location        within the 3D image data, and    -   generate an output, on the output device, that is indicative of        the relationship between the location of the portion of the tool        with respect to the 3D image data and the given location within        the 3D image data.

In some applications, the at least one computer processor is configuredto generate the output that is indicative of the relationship betweenthe location of the portion of the tool with respect to the 3D imagedata and the given location within the 3D image data by generating theoutput upon a 2D cross-section of the skeletal portion that is derivedfrom the 3D image data. In some applications, the at least one computerprocessor is configured to generate the output that is indicative of therelationship between the location of the portion of the tool withrespect to the 3D image data and the given location within the 3D imagedata by generating the output upon a 2D projection of the skeletalportion that is derived from the 3D image data. In some applications,the at least one computer processor is configured to generate the outputthat is indicative of the relationship between the location of theportion of the tool with respect to the 3D image data and the givenlocation within the 3D image data by generating the output upon a 3Dimage of the skeletal portion that is derived from the 3D image data.

In some applications, the at least one computer processor is configuredto derive the relationship between the location of the portion of thetool with respect to the 3D image data and the given location within the3D image data by deriving an anticipated longitudinal insertion path ofthe tool with respect to the given location within the 3D image data. Insome applications, the at least one computer processor is configured toderive the relationship between the location of the portion of the toolwith respect to the 3D image data and the given location within the 3Dimage data by deriving a relationship between the first location of theportion of the tool with respect to the 3D image data and apredesignated target location within the 3D image data. In someapplications, the at least one computer processor is configured toderive the relationship between the location of the portion of the toolwith respect to the 3D image data and the given location within the 3Dimage data by deriving a relationship between the first location of theportion of the tool with respect to the 3D image data with respect torespective volumes within the 3D image data, the respective volumesdesignating respective levels of acceptability of protrusion of the toolwith respect to the respective volumes.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theskeletal portion,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured to sequentially:

-   -   acquire a first 2D x-ray image of at least a portion of the tool        and the skeletal portion from a first view, while the 2D x-ray        imaging device is disposed at a first pose with respect to the        subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least the portion of the tool and        the skeletal portion from a second view, and

(c) an output device,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of the skeletal portion from the 3D        imaging device,    -   receive a designation of a location within the skeletal portion,        with respect to the 3D image data,    -   receive the first and second 2D x-ray images of at least the        portion of the tool and the skeletal portion from the 2D x-ray        imaging device,    -   register the first and second 2D x-ray images to the 3D image        data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second 2D projections that            match the first and second 2D x-ray images of the skeletal            portion,    -   based upon the registration of the first and second 2D x-ray        images to the 3D image data, derive a position of the designated        location within at least one of the 2D x-ray images of the        skeletal portion,    -   identify a location of at least a portion of the tool with        respect to the at least one of the 2D x-ray images of the        skeletal portion, by means of image processing,    -   based upon the identified location of the portion of the tool,        determine, within the at least one of the 2D x-ray images of the        skeletal portion, a relationship between an anticipated        longitudinal insertion path of the tool and the designated        location, and    -   generate an output on the output device, at least partially in        response thereto.

In some applications, the at least one computer processor is configuredto receive the designation of a location within the skeletal portionwith respect to the 3D image data by receiving a designation of a targetlocation within the skeletal portion with respect to the 3D image data.In some applications, the at least one computer processor is configuredto receive the designation of a location within the skeletal portionwith respect to the 3D image data by receiving a designation of one ormore locations that the tool should avoid within the skeletal portionwith respect to the 3D image data.

In some applications, the at least one computer processor is configuredto receive the designation of a location within the skeletal portionwith respect to the 3D image data by receiving a designation of thelocation with respect to a cross-section of the skeletal portion that isderived from the 3D image data. In some applications, the at least onecomputer processor is configured to receive the designation of alocation within the skeletal portion with respect to the 3D image databy receiving a designation of the location with respect to a 2Dprojection of the skeletal portion that is derived from the 3D imagedata. In some applications, the at least one computer processor isconfigured to receive the designation of a location within the skeletalportion with respect to the 3D image data by receiving a designation ofthe location with respect to a 3D image of the skeletal portion that isderived from the 3D image data.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a skeletal portion within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theskeletal portion,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured to sequentially:

-   -   acquire a first 2D x-ray image of at least a portion of the tool        and the skeletal portion from a first view, while the x-ray        imaging device is disposed at a first pose with respect to the        subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least the portion of the tool and        the skeletal portion from a second view, and

(c) a display,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of the skeletal portion from the 3D        imaging device,    -   receive the first and second 2D x-ray images of at least the        portion of the tool and the skeletal portion from the 2D x-ray        imaging device,    -   register the first and second 2D x-ray images to the 3D image        data, the registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying respective first and second 2D projections that            match the first and second 2D x-ray images of the skeletal            portion,    -   identify a location of a tip of the tool with respect to the        first and second x-ray images of the skeletal portion, by means        of image processing,    -   based upon the identified location of the tip of tool with        respect to the first and second x-ray images of the skeletal        portion, and the registration of the first and second x-ray        images to the 3D image data, determine a location of the tip of        the tool with respect to the 3D image data, and    -   in response thereto, drive the display to display a        cross-section of the skeletal portion, the cross-section being        derived from the 3D image data, and corresponding to the        location of the tool tip.

In some applications, the at least one computer processor is furtherconfigured to drive the display to display an indication of a locationof the tool upon the cross-section of the skeletal portion. In someapplications, the at least one computer processor is configured to drivethe display to display the cross-section of the skeletal portion bydriving the display to display a cross-sectional view selected from thegroup consisting of: an axial cross-section, a coronal cross-section, asagittal cross-section, and a cross-sectional view that is based upon adirection of insertion of the tool.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a given vertebra within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of at leasta portion of the subject's spine that contains the given vertebra,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured to sequentially:

-   -   acquire a first 2D x-ray image of at least at least a portion of        the tool and the portion of the subject's spine that contains        the given vertebra from a first view, while the 2D x-ray imaging        device is disposed at a first pose with respect to the subject's        body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of at least a portion of the tool and at        least the portion of the subject's spine that contains the given        vertebra from a second view, and

(c) an output device,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of at least the portion of the        subject's spine that contains the given vertebra from the 3D        imaging device,    -   receive the first and second 2D x-ray images of at least a        portion of the tool and at least the portion of the subject's        spine that contains the given vertebra from the 2D x-ray imaging        device,    -   receive an input that is indicative of a location of the given        vertebra within the 3D image data,    -   automatically determine a location of the given vertebra within        the first and second x-ray images, by means of image processing,    -   register the given vertebra within the first and second 2D x-ray        images to the given vertebra within the 3D image data, the        registering including:        -   generating a plurality of 2D projections of the given            vertebra from the 3D image data, and        -   identifying respective first and second 2D projections that            match the given vertebra within the first and second 2D            x-ray images of the skeletal portion,    -   identify a location of at least the portion of the tool with        respect to the given vertebra within the first and second 2D        x-ray images of the portion of the spine, by means of image        processing,    -   based upon the registration of the given vertebra within the        first and second 2D x-ray images to the given vertebra within        the 3D image data, and the identified location of at least the        portion of the tool, determine the location of at least the        portion of the tool with respect to the given vertebra within        the 3D image data, and    -   generate an output on the output device, in response thereto.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with a radiographic imaging device,an output device, and an instrument having a straight radiopaquecomponent, the apparatus including:

at least one computer processor configured to:

-   -   receive from the radiographic image device, a radiographic image        in which at least a portion the straight radiopaque component of        the instrument is visible,    -   identify the straight radiopaque component of the instrument        within the radiographic image, by means of image processing,    -   at least partially correct distortion in at least a portion of        the radiographic image by deforming the portion of the        radiographic image, such that the straight radiopaque component        of the instrument within the radiographic image appears        straight, and    -   generate an output on the output device, in response thereto.

In some applications, the at least one computer processor is furtherconfigured, based upon the correction applied to a portion of the imagewithin which the straight radiopaque component of the instrumentappeared, to correct an additional portion of the image.

In some applications, the at least one computer processor is furtherconfigured to register the corrected radiographic image to 3D imagedata. In some applications, the radiographic image includes the straightradiopaque component of the instrument and a portion of a body of asubject, and the at least one computer processor is configured toregister the corrected radiographic image to the 3D image data, by:generating a plurality of 2D projections of the portion of the subject'sbody from the 3D image data, and identifying a 2D projection thatmatches the portion of the subject's body within the radiographic image.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use with a procedure in whichinterventions are performed with respect to at least first and secondvertebrae of a spine of a subject, a display, and an imaging deviceconfigured to acquire imaging data of the subject's spine, the apparatusincluding:

at least one computer processor configured to:

-   -   receive the imaging data from the imaging device,    -   generate, upon the display, a spinal roadmap image of at least a        portion of the spine that contains the first and second        vertebra,    -   automatically label vertebra within the spinal roadmap image,    -   determine that an intervention has been performed with respect        to the first vertebra, such that an appearance of the first        vertebra has changed, and    -   automatically update the spinal roadmap to reflect the change in        the appearance of the first vertebra, such that the updated        spinal roadmap is displayed while the intervention is performed        with respect to the second vertebra.

In some applications, the imaging device includes an imaging deviceconfigured to acquire 3D imaging data, and the at least one computerprocessor is configured to generate the spinal roadmap image bygenerating a 3D spinal roadmap image. In some applications, the imagingdevice includes an imaging device configured to acquire 2D imaging data,and the at least one computer processor is configured to generate thespinal roadmap image by generating a 2D spinal roadmap image.

In some applications, the at least one computer processor is configuredto determine that an intervention has been performed with respect to thefirst vertebra, such that an appearance of the first vertebra haschanged by determining that a tool has been inserted into the firstvertebra, and the at least one computer processor is configured toupdate the spinal roadmap by updating the spinal roadmap to display thetool inside the vertebra.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use during a medical intervention inwhich a tool is used with respect to a portion of a body of a subject,and an imaging device that is used to acquire a plurality of images ofthe tool and the subject's body, during the intervention, the apparatusincluding:

at least one computer processor configured to determine a location ofthe tool with respect to the subject's body, by analyzing the pluralityof images;

a motion detection sensor configured to be disposed upon the subject'sbody and to detect motion of at least the portion of the subject's bodyby detecting that its own motion has occurred relative to a priorposition of itself, the motion detection sensor being configured todetect that, between acquisitions of two or more images, motion of atleast a portion of the subject's body that exceeds a threshold amounthas occurred; and

an output device configured to generate an alert indicating that themotion has occurred, in response to the motion detection sensordetecting that motion of at least the portion of the subject's body thatexceeds the threshold amount has occurred.

In some applications, the motion detection sensor is configured to drivethe output device to generate the alert. In some applications, the atleast one computer processor is configured to receive a signal from thesensor indicating that the motion of at least the portion of thesubject's body that exceeds the threshold amount has occurred, and todrive the output device to generate the alert in response thereto. Insome applications, the output device is configured to generate an outputindicating that one or more images should be reacquired, in response tothe motion detection sensor detecting that motion of at least theportion of the subject's body that exceeds the threshold amount hasoccurred.

There is further provided, in accordance with some applications of thepresent invention, apparatus for performing a procedure using a toolconfigured to be advanced into a given vertebra within a body of asubject along a longitudinal insertion path, and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theskeletal portion,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured to acquire one or more 2D x-ray images ofat least at least a portion of the tool and the skeletal portion from asingle view, while the 2D x-ray imaging device is disposed at a firstpose with respect to the subject's body, and while a tip of the tool isdisposed at a given location along the longitudinal insertion path, and

(c) a display,

the apparatus including:

at least one computer processor configured to:

-   -   receive the 3D image data of the skeletal portion from the 3D        imaging device, receive the one or more 2D x-ray images of the        skeletal portion from the 2D x-ray imaging device,    -   register one of the one or more 2D x-ray images that were        acquired from the single view to the 3D image data, the        registering including:        -   generating a plurality of 2D projections from the 3D image            data, and        -   identifying a 2D projection that matches the one of the one            or more 2D x-ray images of the skeletal portion that were            acquired from the single view, identify a location of the            tip of the tool with respect to the skeletal portion within            the one of the one or more 2D x-ray images, by means of            image processing,    -   based upon the identified location of the tip of tool with        respect to the skeletal portion within the one of the one or        more 2D x-ray images, and the registration of the one of the one        or more 2D x-ray images to the 3D image data, determine a        location of the tip of the tool with respect to the 2D        projection that matches the one of the one or more 2D x-ray        images of the skeletal portion, and    -   in response thereto, drive the display to display a        cross-section of the skeletal portion, the cross-section being        derived from the 3D image data, and corresponding to the        location of the tool tip with respect to the 2D projection that        matches the one of the one or more 2D x-ray images of the        skeletal portion.

In some applications, the at least one computer processor is furtherconfigured to drive the display to display a line upon thecross-section, the line indicating that the location of the tool tipwithin the cross-section is somewhere along the line.

There is further provided, in accordance with some applications of thepresent invention, apparatus for use during a medical intervention inwhich a tool is used with respect to a portion of a body of a subject,and for use with:

(a) a 3D imaging device configured to acquire 3D image data of theportion of the subject's body,

(b) a 2D x-ray imaging device that is unregistered with respect to thesubject's body and configured to sequentially:

-   -   acquire a first 2D x-ray image of a distal portion of the tool        and the portion of the subject's body from a first view, while        the 2D x-ray imaging device is disposed at a first pose with        respect to the subject's body,    -   be moved to a second pose with respect to the subject's body,        and    -   while the 2D x-ray imaging device is at the second pose, acquire        a second 2D x-ray image of the distal portion of the tool and        the portion of the subject's body from a second view, and

(c) a display,

the apparatus including:

at least one computer processor configured to:

-   -   calculate a location of the proximal portion of the tool that is        disposed outside the subject's body,    -   based upon the calculated position of the proximal portion of        the tool, derive a location of the distal portion of the tool        with respect to the portion of the subject's body with respect        to the 3D image data,    -   based upon the derived location, drive the display to display an        indication of the location of the distal portion of the tool        with respect to the portion of the subject's body with respect        to the 3D image data,    -   subsequently:        -   receive from the 2D x-ray imaging device the first and            second 2D x-ray images of the distal portion of the tool and            the portion of the subject's body,        -   register the portion of the subject's body within the first            and second 2D x-ray images to the portion of the subject's            body within the 3D image data,        -   identify a location of at least the distal portion of the            tool with respect to the portion of the subject's body            within the first and second 2D x-ray images, by means of            image processing,        -   based upon the registration of the portion of the subject's            body within the first and second 2D x-ray images to the            portion of the subject's body within the 3D image data, and            the identified location of at least the distal portion of            the tool within the first and second 2D x-ray images,            determine the location of at least the distal portion of the            tool with respect to the portion of the subject's body with            respect to the 3D image data, and        -   based upon the determined location of at least the distal            portion of the tool, drive the display to update the            indication of the location of the distal portion of the tool            with respect to the portion of the subject's body with            respect to the 3D image data.

In some applications, the at least one computer processor is configuredto register the portion of the subject's body within the first andsecond 2D x-ray images to the portion of the subject's body within the3D image data, the registering including: generating a plurality of 2Dprojections of the portion of the subject's body from the 3D image data,and identifying respective first and second 2D projections that matchthe portion of the subject's body within the first and second 2D x-rayimages of the portion of the subject's body.

In some applications, the apparatus further includes one or morelocation sensors coupled to the proximal portion of the tool, and the atleast one computer processor is configured to calculate the location ofthe proximal portion of the tool that is disposed outside the subject'sbody by means of the one or more location sensors that are coupled tothe proximal portion of the tool. In some applications, the at least onecomputer processor is configured to calculate the location of theproximal portion of the tool that is disposed outside the subject's bodyby video tracking the proximal portion of the tool. In someapplications, the apparatus further includes a robot, the proximalportion of the tool being coupled to a portion of the robot, and the atleast one computer processor is configured to calculate the location ofthe proximal portion of the tool that is disposed outside the subject'sbody by means of tracking the portion of the robot relative to a priorknown position of the portion of the robot.

The present invention will be more fully understood from the followingdetailed description of embodiments thereof, taken together with thedrawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic illustration of an orthopedic operating room, asused in prior art techniques;

FIG. 1B is a schematic illustration of a system for use with proceduresthat are performed on skeletal anatomy, in accordance with someapplications of the present invention;

FIG. 2 is a schematic illustration of two tools (e.g., Jamshidi™needles) being inserted into a vertebra and the desired insertionwindows for the insertion of such tools, as used in prior arttechniques;

FIGS. 3A and 3B are schematic illustrations of a 3D CT image of avertebra (FIG. 3A), as well as a 2D axial slice that is derived from the3D CT image (FIG. 3B), as used in prior art techniques;

FIGS. 4A and 4B show a C-arm being used to acquire an anterior-posterior(“AP”) 2D radiographic image and a resultant AP image (FIG. 4A), and theC-arm being used to acquire a lateral 2D radiographic image and aresultant lateral image (FIG. 4B), as used in prior art techniques;

FIGS. 5A, 5B, 5C, 5D, and 5E are schematic illustration of sets ofradiopaque markers which are placed upon a subject, in accordance withsome applications of the present invention;

FIGS. 6A and 6B are schematic illustrations of first and second sets ofradiopaque markers configured to be placed on a subject, in accordancewith some applications of the present invention;

FIG. 7 is a flowchart showing steps that are typically performed usingthe system of FIG. 1B, in accordance with some applications of thepresent invention;

FIG. 8A shows a vertebra designated upon cross-sectional images of asubject's spine that are derived from 3D image data, in accordance withsome applications of the present invention;

FIG. 8B shows an example of a 3D CT image of a subject's spine displayedalongside a combined radiographic image of the subject's spine, inaccordance with some applications of the present invention;

FIG. 8C shows the designated vertebra indicated on a 3D CT image and ona 2D x-ray image, the CT image and x-ray image being displayed alongsideone another, in accordance with some applications of the presentinvention;

FIG. 9 shows an example of an optical image displayed alongside a 2Dradiographic image, in accordance with some applications of the presentinvention;

FIG. 10 shows an example of a 2D radiographic (e.g., x-ray) imagedisplayed alongside a cross-sectional image of a subject's vertebra thatis derived from 3D image data of the vertebra, in accordance with someapplications of the present invention;

FIGS. 11A and 11B show examples of respectively AP and lateral x-rayimages of a Jamshidi™ needle being inserted into a subject's spine, inaccordance with some applications of the present invention;

FIGS. 12A and 12B show examples of correspondence between respectiveviews of a 3D image a vertebra, with corresponding respective first andsecond x-ray images of the vertebra, in accordance with someapplications of the present invention;

FIGS. 13A, 13B, and 13C are schematic illustrations that demonstrate therelationship between a 3D image of an object (which in the example shownin FIG. 13A is a cone) and side-to-side (FIG. 13B) and bottom-to-top(FIG. 13C) 2D projection images of the object, such a relationship beingutilized, in accordance with some applications of the present invention;

FIGS. 14A and 14B are flowcharts showing steps that are typicallyperformed using the system of FIG. 1B, in accordance with someapplications of the present invention;

FIG. 15A shows an example of axial cross-sections of a vertebracorresponding, respectively, to first and second locations of a tip of atool that is advanced into the vertebra along a longitudinal insertionpath, as shown on corresponding 2D x-ray images that are acquired from asingle x-ray image view, in accordance with some applications of thepresent invention;

FIG. 15B shows an example of axial cross-sections of a vertebra uponwhich, respectively, first and second locations of a tip of a tool thatis advanced into the vertebra along a longitudinal insertion path aredisplayed, the locations being derived using x-ray images acquired fromtwo or more x-ray image views, in accordance with some applications ofthe present invention;

FIGS. 16A and 16B show examples of a display showing a given locationdesignated upon 3D (e.g., CT or MRI) image data and a relationshipbetween an anticipated longitudinal insertion path of a tool and thegiven location upon, respectively, AP and lateral 2D x-ray images, inaccordance with some applications of the present invention;

FIG. 17A shows an AP x-ray of two tools being inserted into a vertebrathrough, respectively, 10-11 o'clock and 1-2 o'clock insertion windows,the AP x-ray being generated using prior art techniques;

FIG. 17B shows a corresponding lateral x-ray image to FIG. 17A, thelateral x-ray being generated using prior art techniques;

FIG. 18 is a schematic illustration of a Jamshidi™ needle with aradiopaque clip attached thereto, in accordance with some applicationsof the present invention;

FIG. 19A shows an AP x-ray image and a corresponding lateral x-ray imageof a vertebra, at a first stage of the insertion of a tool into thevertebra, in accordance with some applications of the present invention;

FIG. 19B shows an AP x-ray image of the vertebra, at a second stage ofthe insertion of the tool into the vertebra, and an indication of thederived current location of the tool tip displayed upon a lateral x-rayimage of the vertebra, in accordance with some applications of thepresent invention;

FIG. 20 is a schematic illustration of a three-dimensional rigid jigthat comprises at least portions thereof that are radiopaque andfunction as radiopaque markers, the radiopaque markers being disposed ina predefined three-dimensional arrangement, in accordance with someapplications of the present invention;

FIG. 21A show examples of x-ray images of a calibration jig generated bya C-arm that uses an image intensifier, and by a C-arm that uses aflat-panel detector, such images reflecting prior art techniques; and

FIG. 21B shows an example of an x-ray image acquired by a C-arm thatuses an image intensifier, the image including a radiopaque componentthat corresponds to a portion of a tool that is known to be straight,and a dotted line overlaid upon the image indicating how a line definedby the straight portion would appear if distortions in the image arecorrected, in accordance with some applications of the presentinvention.

DETAILED DESCRIPTION OF EMBODIMENTS

Reference is now made to FIG. 1B, which is a schematic illustration of asystem 20 for use with procedures that are performed on skeletalanatomy, in accordance with some applications of the present invention.For some applications, the system is used for a procedure that isperformed on one or more vertebrae of a subject's spine. However, thescope of the present invention includes applying any of the apparatusand methods described herein to procedures performed on other portionsof a subject's skeletal anatomy, mutatis mutandis. Such procedures mayinclude joint (e.g., shoulder, knee, hip, and/or ankle) replacement,joint repair, fracture repair (e.g., femur, tibia, and/or fibula), aprocedure that is performed on a rib (e.g., rib removal, or ribresection), and/or other interventions in which 3D image data areacquired prior to the intervention and 2D images are acquired during theintervention.

System 20 typically includes a computer processor 22, which interactswith a memory 24, and one or more user interface device 26. Typically,the user interface devices include one or more input devices, such as akeyboard 28 (as shown), and one or more output devices, e.g., a display30, as shown. Inputs to, and outputs from, the computer processor thatare described herein are typically performed via the user interfacedevices. For some applications, the computer processor as well as thememory and the user interface devices, are incorporated into a singleunit, e.g., a tablet device, and/or a laptop computer.

For some applications, the user interface devices include a mouse, ajoystick, a touchscreen device (such as a smartphone or a tabletcomputer), a touchpad, a trackball, a voice-command interface, and/orother types of user interfaces that are known in the art. For someapplications, the output device includes a head-up display and/or ahead-mounted display, such as Google Glass®. For some applications, thecomputer processor generates an output on a different type of visual,text, graphics, tactile, audio, and/or video output device, e.g.,speakers, headphones, a smartphone, or a tablet computer. For someapplications, a user interface device acts as both an input device andan output device. For some applications, computer processor 22 generatesan output on a computer-readable medium (e.g., a non-transitorycomputer-readable medium), such as a disk or a portable USB drive. Forsome applications, the computer processor comprises a portion of apicture archiving and communication system (PACS), and is configured toreceive inputs from other components of the system, e.g., via memory 24.Alternatively or additionally, the computer processor is configured toreceive an input on a computer-readable medium (e.g., a non-transitorycomputer-readable medium), such as a disk or a portable USB drive. It isnoted that, for some applications, more than one computer processor isused to perform the functions described herein as being performed bycomputer processor 22.

Typically, 3D image data are acquired before the subject is in theoperating room for the procedure, or when the subject is in theoperating room, but before an intervention has commenced. For example,3D CT image data of the portion of the skeletal anatomy upon which theprocedure is to be performed (and/or neighboring portions of theanatomy) may be acquired using a CT scanner 32. Alternatively oradditionally, 3D MRI image data of the portion of the skeletal anatomyupon which the procedure is to be performed (and/or neighboring portionsof the anatomy) may be acquired using an MRI scanner. For someapplications, 3D x-ray data are acquired. Typically, the 3D image dataare transferred to memory 24, and are retrieved from the memory bycomputer processor 22. It is noted that for illustrative purposes, FIG.1B shows the CT scanner, the C-arm, and system 20 together with oneanother. However, in accordance with the above description, for someapplications, the CT scanner is not disposed in the same room as system20, and/or C-arm 34.

During the procedure, real time 2D images are acquired by a radiographicimaging device, e.g., a C-arm 34 (as shown), which acquires 2D x-rayimages. For some applications, the 2D images are captured in real timeby a frame grabber of system 20 that is connected to an output port ofthe C-arm. Alternatively or additionally, system 20 and the C-arm areconnected to one another via a PACS network to which system 20 and C-arm34 are connected, and the 2D images are transferred, once acquired, tosystem 20 via the PACS network (e.g., via memory 24). Alternatively oradditionally, the C-arm sends image files, for example in DICOM format,directly to system 20 (e.g., via memory 24).

Typically, the interventional part of a procedure that is performed onskeletal anatomy, such as the spine, commences with the insertion of atool, such as a Jamshidi™ needle 36. A Jamshidi™ needle typicallyincludes an inner tube and an outer tube. The Jamshidi™ needle istypically inserted to a target location, at which point other toolsand/or implants are inserted using the Jamshidi™ needle. Typically, inopen surgery, for lower-diameter tools and/or implants, the inner tubeof the Jamshidi™ needle is removed, and the tool and/or implant isinserted via the outer tube of the Jamshidi™ needle, while forlarger-diameter tools and/or implants, the tool and/or implant isinserted by removing the inner tube of the Jamshidi™ needle, inserting astiff wire through the outer tube, removing the outer tube, and theninserting the tool and/or implant along the stiff wire. Forminimally-invasive surgery, the aforementioned steps (or similar stepsthereto) are typically performed via small incisions.

It is noted that, in general throughout the specification and the claimsof the present application, the term “tool” should be interpreted asincluding any tool or implant that is inserted into any portion of theskeletal anatomy during a procedure that is performed upon the skeletalanatomy. Such tools may include flexible, rigid and/or semi-rigidprobes, and may include diagnostic probes, therapeutic probes, and/orimaging probes. For example, the tools may include Jamshidi™ needles,other needles, k-wires, pedicle markers, screws, nails, other implants,implant delivery probes, drills, endoscopes, probes inserted through anendoscope, tissue ablation probes, laser probes, balloon probes,injection needles, tissue removal probes, drug delivery probes,stimulation probes, dilators, etc. Typically, such procedures includespinal stabilization procedures, such as vertebroplasty (i.e., injectionof synthetic or biological cement in order to stabilize spinalfractures), kyphoplasty (i.e., injection of synthetic or biologicalcement in order to stabilize spinal fractures, with an additional stepof inflating a balloon within the area of the fracture prior toinjecting the cement), fixation (e.g., anchoring two or more vertebraeto each other by inserting devices such as screws into each of thevertebrae and connecting the screws with rods), fixation and fusion(i.e., fixation with the additional step of an implant such as a cageplaced in between the bodies of the vertebrae), and/or endoscopy (i.e.,inserting an endoscope toward a vertebra and/or a disc, for example, inorder to remove tissue (e.g., disc tissue, or vertebral bone) thatcompresses nerves).

Reference is now made to FIG. 2, which is a schematic illustration oftwo Jamshidi™ needles 36 being inserted into a vertebra 38, as used inprior art techniques. Typically, a spinal intervention aimed at avertebral body is performed with tools being aimed at 10-11 o'clock and1-2 o'clock insertion windows with respect to the subject's spine. Toolinsertion into a vertebra must avoid the spinal cord 42, andadditionally needs to avoid exiting the vertebra from the sides, leavingonly two narrow insertion windows 44, on either side of the vertebra. Asdescribed hereinbelow with reference to FIGS. 3A-4B, typically the mostimportant images for determining the locations of the insertion windowsare those derived from 3D image data, and are not available from thereal time 2D images that are typically acquired during the intervention.

Reference is now made to FIGS. 3A and 3B, which are schematicillustrations of a 3D CT image of a vertebra (FIG. 3A), as well as a 2Daxial slice that is derived from the 3D CT image (FIG. 3B), such imagesbeing used in prior art techniques. Reference is also made to FIGS. 4Aand 4B, which show C-arm 34 being used to acquire an anterior-posterior(“AP”) 2D radiographic image and a resultant AP image (FIG. 4A), andC-arm 34 being used to acquire a lateral 2D radiographic image and aresultant lateral image (FIG. 4B), as used in prior art techniques.

As may be observed, the view of the vertebra that is important fordetermining the entry point, insertion direction, and insertion depth ofthe tool is shown in the axial 2D image slice of FIG. 3B. By contrast,the 2D radiographic images that are acquired by the C-arm are summationsof 3D space, and do not show cross-sectional views of the vertebra. Asdescribed hereinabove, Computer Aided Surgery (CAS) systems typicallymake use of CT and/or MRI images, generated before the subject has beenplaced in the operating room, or once the subject has been placed in theoperating room but typically before an intervention has commenced.However, such procedures are typically more expensive than non-CASprocedures (such non-CAS procedures, including open procedures,mini-open procedures, and minimally-invasive procedures), limit toolselection to those fitted with location sensors as described above, andrequire such tools to be individually identified and calibrated at thebeginning of each surgery.

In accordance with some applications of the present invention, theintra-procedural location of a tool is determined with respect to 3Dimage data (e.g., a 3D image, a 2D cross-section derived from 3D imagedata, and/or a 2D projection image derived from 3D image data), in anon-CAS procedure (e.g., in an open, mini-open and/or minimally-invasiveprocedure). The techniques described herein are typically practicedwithout requiring the fitting of location sensors (such as infraredtransmitters or reflectors, or magnetic or electromagnetic sensors) tothe tool or to the subject, and without requiring identification and/orcalibration of tools prior to the procedure. The techniques describedherein typically do not require tracking the location of the subject'sbody or the applicable portion of the subject's body, and do not assumeany knowledge of the location coordinates of the subject's body in somereference frame. The techniques described herein typically do notrequire location sensors that rely upon tracking technologies (e.g.,electromagnetic or IR tracking technologies) that are not typically usedin an orthopedic operating room. Further typically, the techniquesdescribed herein are practiced without requiring knowledge of theprecise parameters of any individual pose of the 2D radiographic imagingdevice (e.g., C-arm 34), and without requiring poses of the 2Dradiographic imaging device (e.g., C-arm 34) to be tracked relative toeach other, and/or relative to the position of the subject. For someapplications, 2D radiographic images (e.g., 2D x-ray images) areacquired from two or more views, by moving a radiographic imaging deviceto respective poses between acquisitions of the images of respectiveviews. Typically, a single x-ray source is used for acquisition of the2D x-ray images, although, for some applications, multiple sources areused. In general, where views of the 2D radiographic imaging device aredescribed herein as being AP, lateral, oblique, etc., this should not beinterpreted as meaning that images must be acquired from precisely suchviews, rather acquiring images from generally such views is typicallysufficient. Typically, the techniques described herein are tool-neutral,i.e., the techniques may be practiced with any applicable tool andtypically without any modification and/or addition to the tool.

It is noted that although some applications of the present invention aredescribed with reference to 3D CT imaging, the scope of the presentinvention includes using any 3D imaging, e.g., MRI, 3D x-ray imaging,and/or other modalities of 3D imaging, mutatis mutandis. Such imagingmay be performed prior to, at the commencement of, and/or at some pointduring, an intervention. For example, the 3D imaging may be performedbefore the subject has been placed within the operating room, when thesubject is first placed within the operating room, or at some point whenthe subject is in the operating room, but prior to the insertion of agiven tool into a given target portion. Similarly, although someapplications of the present invention are described with reference to 2Dradiographic or x-ray imaging, the scope of the present inventionincludes using any 2D imaging, e.g., ultrasound and/or other modalitiesof 2D imaging, mutatis mutandis. Although some applications of thepresent invention are described with reference to procedures that areperformed on skeletal anatomy and/or vertebrae of the spine, the scopeof the present invention includes applying the apparatus and methodsdescribed herein to other orthopedic interventions (e.g., a joint (e.g.,shoulder, knee, hip, and/or ankle) replacement, joint repair, fracturerepair (e.g., femur, tibia, and/or fibula), a procedure that isperformed on a rib (e.g., rib removal, or rib resection), vascularinterventions, cardiovascular interventions, neurovascularinterventions, abdominal interventions, therapeutic irradiations, and/orinterventions performed on other portions of a subject, includinginterventions in which 3D image data are acquired prior to theintervention and 2D images are acquired during the intervention, mutatismutandis.

Reference is now made to FIGS. 5A, 5B, 5C, 5D, and 5E, which areschematic illustration of sets 50 of radiopaque markers 52 which aretypically placed on a subject, in accordance with some applications ofthe present invention. For some applications, the sets of markers aredisposed on a drape 53, as shown. Drape 53 is typically sterile anddisposable. For some applications, the set of markers includes anauthentication and/or an anti-copying element, such as RFID, barcode(s), etc.

Typically, sets 50 of markers 52 are placed on or near the subject in avicinity of a site at which an intervention is to be performed, and suchthat at least some of the markers appear in 2D radiographic images thatare acquired of the intervention site from typical imaging views forsuch an intervention. For example, for a procedure that is performed onthe subject's spine, the markers are typically placed on the subject'sback in a vicinity of the site of the spinal intervention, such that atleast some of the markers appear in 2D radiographic images that areacquired of the intervention site from AP imaging views (as shown inFIGS. 5B, 5C, 5D, and 5E). For some applications, the markers are placedon the subject's side in a vicinity of the site of the spinalintervention, such that at least some of the markers appear in 2Dradiographic images that are acquired of the intervention site from alateral imaging view (FIG. 5A). For some applications, the markers areplaced on the subject's back, such that at least some of the markers arelevel with the subject's sacrum. As shown the markers may be arranged ina line (e.g., like a ruler, FIG. 5A), a grid (FIG. 5B), a pair ofparallel lines (FIG. 5C), a frame around a surgical site (e.g., a circle(FIG. 5D), or a rectangle (FIG. 5E)), and/or any other shape. For someapplications, the markers include one or more meshes, and/or a set ofdiscrete elements that are connected (virtually or physically) into oneor more meshes. For some applications, the set of markers comprises anarrangement wherein portions thereof are visible from different imageviews, for example, the arrangement may include two rulers one of whichis positioned on the subject's back (e.g., as shown in FIG. 5C), such asto be visible in an image acquired from an AP view, and the other one ofwhich is positioned on the subject's side (e.g., as shown in FIG. 5A),such as to be visible in an image acquired from a lateral view. For someapplications (not shown), the set of markers is a rigid arrangement ofidentifiable radiopaque features, for example, a notched radiopaqueruler, placed on or attached to the surgical table.

Typically, surgery on skeletal anatomy commences with attaching asterile surgical drape at and around the surgical site. In the case ofspinal surgery, the surgical approach may be anterior, posterior,lateral, oblique, etc., with the surgical drape placed accordingly. Forsuch applications, sets 50 of markers 52 are typically placed above thesurgical drape. Alternatively, sets of markers are placed on thesubject's skin (e.g., if no surgical drape is used). For someapplications, sets of markers are placed under the subject's body, on(e.g., attached to) the surgical table, and/or such that some of themarkers are above the surgical table in the vicinity of the subject'sbody. For some applications, a plurality of sets of markers are used.For example, multiple sets of markers may be placed adjacently to oneanother. Alternatively or additionally, one or more sets of markers maybe placed on the subject's body such that at least some markers arevisible in each of a plurality of x-ray image views, e.g., on the backor stomach and/or chest for the AP or PA views, and on the side of thebody for the lateral view. For some applications, a single drape 53 withmarkers disposed thereon extends, for example, from the back to theside, such that markers are visible in both AP and lateral x-ray imageviews.

Reference is also made to FIGS. 6A and 6B, which are schematicillustrations of first and second sets 50A and 50B of radiopaque markers52 configured to be placed on a subject, in accordance with someapplications of the present invention. For some applications, the firstand second sets of radiopaque markers are coupled to one another via theone or more surfaces 54. For example, the one or more surfaces maycomprise a portion of drape 53. The one or more surfaces are configuredto position the first and second sets of markers on respective sides ofthe subject's spine, in predefined positions with respect to one another(e.g., parallel and/or with one or both ends of each of the sets alignedwith one another), by the one or more surfaces being placed over aportion of the subject's spine upon which the procedure is to beperformed. Subsequent to positioning the first and second set of markerson the respective sides of the subject's spine, the one or more surfacesare configured to be removable from the subject's body (for example, thesurfaces may be peeled off), such as to facilitate performance of theprocedure upon the portion of the subject's spine over which the one ormore surfaces were placed, while leaving the first and second sets ofmarkers on the respective sides of the subject's spine, in theirpredefined positions with respect to one another.

Typically, the sets of markers are thereby positioned in a desiredrelationship with respect to one another (e.g., parallel and/or with oneor both ends of each of the sets aligned with one another). However, byvirtue of removing surface(s) 54, fragments of the surface(s) areprevented from entering the body, and/or interfering with theintervention. Typically, the sets of markers are positioned on eitherside of the subject's spine such that even in oblique x-ray image viewsof the intervention site (and neighboring portions of the spine), atleast markers belonging to one of the sets of markers are visible.Further typically, the sets of markers are positioned on either side ofthe subject's spine such that even in zoomed-in views acquired from thedirection of the tool insertion, or in views that are oblique (i.e.,diagonal) relative to the direction of tool insertion, at least markersbelonging to one of the sets of markers are visible. Typically, the setsof radiopaque markers are placed on the subject, such that theradiopaque markers do not get in the way of either AP or lateral x-rayimages of vertebrae, such that the radiopaque markers do not interferewith the view of the surgeon during the procedure, and do not interferewith registration of 2D and 3D image data with respect to one another(which, as described hereinbelow, is typically based on geometry of thevertebrae).

Typically, the sets of markers as shown in FIG. 6A are used inopen-surgery procedures. For such procedures, a relatively large centralwindow is required for performing the procedure between the two sets ofmarkers.

Typically, minimally-invasive spinal interventions are performed viasmall incisions aimed at 10-11 o'clock and 1-2 o'clock insertionwindows. For some such applications, an additional marker set 50C isplaced along a site that is between the 11 o'clock and 1 o'clockpositions, as shown in FIG. 6B. The additional set of markers ispositioned with respect to the first and second sets of markers, suchthat the additional set of markers is placed along a center of thesubject's spine, when the first and second sets of markers are placedalong respective sides of the spine. For such applications, whensurfaces 54 are removed, an intervention is typically performed viaincisions aimed at 10-11 o'clock and 1-2 o'clock insertion windows,which are disposed, respectively, between marker sets 50A and 50C, andbetween marker sets 50B and 50C.

Radiopaque markers 52 typically include markings (e.g., lines, notches,numbers, characters, shapes) that are visible to the naked eye as wellas to the imaging that is applied. Typically, the markers are radiopaquesuch that the markers are visible in radiographic images. Furthertypically, markers that are placed at respective locations with respectto the subject are identifiable. For example, as shown in FIGS. 6A and6B respective radiopaque alphanumeric characters are disposed atrespective locations. For some applications, markers placed atrespective locations are identifiable based upon other features, e.g.,based upon the dispositions of the markers relative to other markers.Using a radiographic imaging device (e.g., C-arm 34), a plurality ofradiographic images of the set of radiopaque markers are acquired,respective images being of respective locations along at least a portionof the subject's spine and each of the images including at least some ofthe radiopaque markers.

As shown in FIGS. 6A-6B, typically, each of the sets of markers includesradiopaque notches. For some applications, to facilitate interpretationof the images and, for example, to avoid confusion when only one ruleris visible, each of the sets of markers is additionally marked with arespective set of radiopaque characters, the sets being different fromone another. For example, as shown, set 50A is marked with numbers andset 50B is marked with letters.

Typically, drape 53 is made of a flexible material that is configured togenerally conform to contours of the subject's body. For someapplications, at least some of radiopaque markers 52 are rigid and haverespective known dimensions. For example, markers 58 shown in FIGS. 6Aand 6B may be rigid. The rigid radiopaque markers are disposed upon theflexible material, such that, typically, rigidity of the rigid markersdoes not prevent the flexible material from generally conforming to thecontours of the subject's body. Typically, computer processor 22identifies, by means of image processing, at least one of the rigidradiopaque markers within a radiographic image of the spine acquired bythe radiographic imaging device (e.g., C-arm 34), and based upon theidentified rigid radiopaque markers, determines dimensions of featureswithin the radiographic image (e.g., features that were automaticallyidentified, and/or features designated by an operator of system 20), bymeans of image processing. In accordance with respective applications,the rigid markers are 2D or 3D, and are radiopaque in whole or in part.

For some applications, rigid radiopaque markers (and/or a rigidradiopaque jig) that appear in a plurality of different in x-ray imageviews of the subject are used to aid registering x-ray images to oneanother, in general accordance with techniques described herein. Forsome applications, rigid radiopaque markers (and/or a rigid radiopaquejig) that appear in a plurality of different in x-ray image views of thesubject are used to aid registering x-ray images to 3D image data (e.g.,to CT image data) and/or to one another, in general accordance withtechniques described herein.

For some applications, an arrangement similar to the arrangement shownand described with reference to FIG. 6A is applied to first and secondsets of radiopaque markers, the first set 50A being positioned such thatit is visible from the intended direction of intervention (for example,dorsal) corresponding to a first anticipated view of the x-ray imaging(for example, AP) and the second set 50B being positioned such that itis visible from a secondary direction (for example, lateral)corresponding to a second anticipated view of the x-ray imaging (forexample, a left lateral view or a right lateral view). The spatialrelationship of the marker sets to one another is known, for example, itmay be defined by the one or more surfaces 54 via which the marker setsare coupled to each other. (For some applications, such surfaces areremovable as described hereinabove.) Once applied to the subject, thetwo marker sets 50A and 50B typically facilitate the association ofspecific vertebrae, as seen in one x-ray view (for example, an AP view),with same vertebrae as seen in a second x-ray view (for example, a leftlateral or a right lateral view). For some applications, suchassociation is performed manually by the surgeon referring to theradiopaque markers, e.g., by the surgeon identifying markers that have aknown association with one another in the x-ray images, e.g., viamatching of alphanumeric characters. Alternatively or additionally, theassociation is performed automatically by computer processor 22 ofsystem 20 by means of image processing, such as by identifying markersthat have a known association with one another in the x-ray images,e.g., via pattern matching, or via optical character recognition.

For some applications, sets 50 of markers 52, and/or a rigid radiopaquejig, as described hereinabove, are used to facilitate any one of thefollowing functionalities:

-   -   Vertebra level verification, as described hereinbelow.    -   Arriving at a desired vertebra intra-procedurally, without        requiring needles to be stuck into the patient, and/or counting        along a series of non-combined x-rays.    -   Displaying a 3D image of the spine that includes indications of        vertebra thereon, using vertebral level verification.    -   Determining the correct incision site(s) prior to actual        incision(s).    -   Providing a reference scale with some known        dimensions/intervals, e.g., by using rigid markers 58 to provide        reference dimensions, and/or by comparing known shapes and/or        sizes, e.g. of rigid markers 58, to what is seen in an image        (e.g., in order to determine the extent of foreshortening in the        image).    -   Performing measurements on images of the subject, e.g., by using        the rigid markers to provide reference dimensions, and/or by        comparing known shapes and/or sizes, e.g. of the rigid markers        58, to what is seen in an image (e.g., in order to determine the        extent of foreshortening in the image).    -   Registration of 2D radiographic images to 3D image data. In this        regard, it is noted that if the 3D image data were acquired in        the operating room, then markers 52 and rigid markers 58 may be        observable in the 3D image data and at least some of the 2D        images, which typically further facilitates registration. If the        3D image data were acquired previously, the 3D imaging data        typically includes some built-in dimensions, which the 2D image        data typically do not. Rigid markers 58 provide reference        dimensions in the 2D images, which for some applications may be        used as an additional input to register the 2D images to the 3D        image data.    -   Identifying changes in a pose of the 2D imaging device (e.g.,        the x-ray C-arm) and/or a position of the patient. Typically, if        the position of the 2D imaging device relative to the subject,        or the position of the subject relative to the 2D imaging        device, has changed, then in the 2D images there would be a        visible change in the appearance of the markers 52 relative to        the anatomy within the image. For some applications, in response        to detecting such a change, the computer processor generates an        alert. Alternatively or additionally, the computer processor may        calculate the change in position, and account for the change in        position, e.g., in the application of algorithms described        herein. Further alternatively or additionally, the computer        processor assists the surgeon in returning the 2D imaging device        to a previous position relative to the subject. For example, the        computer processor may generate directions regarding where to        move an x-ray C-arm, in order to replicate a prior imaging        position, or the computer processor may facilitate visual        comparison by an operator.    -   Providing a reference for providing general orientation to the        surgeon throughout a procedure.    -   Providing information to the computer processor regarding the        orientation of image acquisition and/or tool insertion, e.g.,        anterior-posterior (“AP”) or posterior-anterior (“PA”), left        lateral or right lateral, etc.    -   Generating and updating a visual roadmap of the subject's spine,        as described in further detail hereinbelow.

For some applications, at least some of the functionalities listed aboveas being facilitated by use of sets 50 of markers 52, and/or a rigid jigare performed by computer processor 22 even in the absence of sets 50 ofmarkers 52, and/or a rigid jig, e.g., using techniques as describedherein. Typically, sets 50 of markers 52, and/or a rigid jig are usedfor level verification, the determination of a tool entry point or anincision site, performing measurements using rigid markers as areference, identifying changes in a relative pose of the 2D imagingdevice (e.g., the x-ray C-arm) and of the subject, and providing generalorientation. All other functionalities of system 20 (such asregistration of 2D images to 3D image data and other functionalitiesthat are derived therefrom) typically do not necessarily require the useof sets 50 of markers 52, and/or a rigid jig. The above-describedfunctionalities may be performed automatically by computer processor 22,and/or manually.

Applications of the present invention are typically applied, in non-CASspinal surgery, to one or more procedural tasks including, withoutlimitation:

-   -   Applying pre-operative 3D visibility (e.g., from CT and/or MRI)        during the intervention. It is noted that 3D visibility provides        desired cross-sectional images (as described in further detail        hereinbelow), and is typically more informative and/or of better        quality than that provided by intraoperative 2D images. (It is        noted that, for some applications, intraoperative 3D imaging is        performed.)    -   Confirming the vertebra(e) to be operated upon.    -   Determining the point(s) of insertion of one or more tools.    -   Determining the direction of insertion of one or more tools.    -   Monitoring tool progression, typically relative to patient        anatomy, during insertion.    -   Reaching target(s) or target area(s).    -   Exchanging tools while repeating any of the above steps.    -   Determining tool/implant position within the anatomy, including        in 3D.    -   Generating and updating a visual roadmap of the subject's spine,        as described in further detail hereinbelow.

Reference is now made to FIG. 7, which is a flowchart showing steps thatare typically performed using system 20, in accordance with someapplications of the present invention. It is noted that some of thesteps shown in FIG. 7 are optional, and some of the steps may beperformed in a different order to that shown in FIG. 7. In a first step70, targeted vertebra(e) are marked by an operator with respect to 3Dimage data (e.g., a 3D image, a 2D cross-section derived from 3D imagedata, and/or a 2D projection image derived from 3D image data) of thesubject's spine. For some applications, in a second step 72, sets 50 ofmarkers 52 are placed on the subject, underneath the subject, on thesurgical table, or above the surgical table in a vicinity of thesubject. For some applications, step 72 is performed prior to step 70.Typically, in a third step 74, vertebrae of the spine are identified inorder to verify that the procedure is being performed with respect tothe correct vertebra (a step which is known as “level verification”),using radiographic images of the spine and the markers to facilitate theidentification. In a fourth step 76, an incision site (in the case ofminimally-invasive surgery) or a tool entry point (in the case of opensurgery) is determined. In a fifth step 78, the first tool in thesequence of tools (which is typically a needle, e.g., a Jamshidi™needle) is typically inserted into the subject (e.g., in the subject'sback), and is slightly fixated in the vertebra. In a sixth step 80, twoor more 2D radiographic images are acquired from respective views thattypically differ by at least 10 degrees (and further typically by 30degrees or more), and one of which is typically from the direction ofinsertion of the tool. Typically, generally-AP and generally-lateralimages are acquired. Alternatively or additionally, images fromdifferent views are acquired. In a seventh step 82, computer processor22 of system 20 typically registers the 3D image data to the 2D images.

Subsequent to the registration of the 3D image data to the 2D imagesadditional features of system 20 as described in detail hereinbelow maybe applied by computer processor 22. For example, in step 84, thecomputer processor drives display 30 to display a cross-section derivedfrom the 3D image data at a current location of the tip of a tool asidentified from a 2D image, and, optionally, to show a vertical line onthe cross-sectional image indicating a line within the cross-sectionalimage somewhere along which the tip of the tool is currently disposed.

It is noted, that, as described in further detail hereinbelow, for someapplications, in order to perform step 84, the computer processor needacquire one or more 2D x-ray images of a tool at a first location insidethe vertebra from only a single x-ray image view, and the one or more 2Dx-ray images are registered to the 3D image data by generating aplurality of 2D projections from the 3D image data, and identifying a 2Dprojection that matches the 2D x-ray images of the vertebra. In responseto registering the one or more 2D x-ray images acquired from the singlex-ray image view to the 3D image data, the computer processor drives adisplay to display a cross-section derived from the 3D image data at athe first location of a tip of the tool, as identified from the one ormore 2D x-ray images, and optionally to show a vertical line on thecross-sectional image indicating a line within the cross-sectional imagesomewhere along which the first location of the tip of the tool isdisposed. Typically, when the tip of the tool is disposed at anadditional location with respect to the vertebra, further 2D x-rayimages of the tool at the additional location are acquired from the samesingle x-ray image view, or a different single x-ray image view, and theabove-described steps are repeated. Typically, for each location of thetip of the tool to which the above-described technique is applied, 2Dx-ray images need only be acquired from a single x-ray image view, whichmay stay the same for the respective locations of the tip of the tool,or may differ for respective locations of the tip of the tool.Typically, two or more 2D x-rays are acquired from respective views, andthe 3D image data and 2D x-ray images are typically registered to the 3Dimage data (and to each other) by identifying a corresponding number of2D projections of the 3D image data that match respective 2D x-rayimages. In step 86, the computer processor drives display 30 to displaythe anticipated (i.e., extrapolated) path of the tool with reference toa target location and/or with reference to a desired insertion vector.In step 88, the computer processor simulates tool progress within asecondary 2D imaging view, based upon observed progress of the tool in aprimary 2D imaging view. In step 90, the computer processor overlays animage of the tool, a representation thereof, and/or a representation ofthe tool path upon the 3D image data (e.g., a 3D image, a 2Dcross-section derived from 3D image data, and/or a 2D projection imagederived from 3D image data), the location of the tool or tool pathhaving been derived from current 2D images.

Reference is now made to FIG. 8A, which shows a vertebra 91 designatedupon a coronal cross-sectional image 92 and upon a sagittalcross-sectional image 94 of a subject's spine, the cross-sectionalimages being derived from 3D image data, in accordance with someapplications of the present invention. As described hereinabove withreference to step 70 of FIG. 7, typically prior to the subject beingplaced into the operating room, or while the subject is in the operatingroom but before an intervention has commenced, an operator marks thetargeted vertebra(e) with respect to the 3D image data (e.g., a 3Dimage, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data). For some applications, inresponse to the operator marking one vertebra, the computer processordesignates additional vertebra(e). For some applications, the operatormarks any one of, or any combination of, the following with respect tothe 3D image data: a specific target within the vertebra (such as afracture, a tumor, etc.), desired approach directions/vectors for toolinsertion, and/or desired placement locations of implants (such aspedicle screws). For some applications, the operator marks the targetedvertebra with respect to a 2D x-ray image that has a sufficiently largefield of view to encompass an identifiable portion of the anatomy (e.g.,the sacrum) and the targeted vertebra(e). For some applications, morethan one targeted vertebra is marked, and for some applications, two ormore vertebra(e) that are not adjacent to one another are marked.

For some applications, the computer processor automatically counts thenumber of vertebrae on the image from an identifiable anatomicalreference (e.g., the sacrum) to the marked target vertebra(e). It isthen known that the targeted vertebra(e) is vertebra N from theidentifiable anatomical reference (even if the anatomical labels of thevertebra(e) are not known). For some applications, the vertebra(e) arecounted automatically using image-processing techniques. For example,the image-processing techniques may include shape recognition ofanatomical features (of vertebrae as a whole, of traverse processes,and/or of spinous processes, etc.). Or, the image-processing techniquesmay include outer edge line detection of spine (in a 2D image of thespine) and then counting the number of bulges along the spine (eachbulge corresponding to a vertebra). For some applications, theimage-processing techniques include techniques described in US2010-0161022 to Tolkowsky, which is incorporated herein by reference.

Referring to step 72 of FIG. 7 in more detail, for some applications, inwhich a procedure is performed on a given vertebra of the subject'sspine, one or more sets 50 of radiopaque markers 52 are placed upon ornear the subject, such that markers that are placed at respectivelocations with respect to the subject are identifiable, e.g., as shownin FIGS. 5A-E and 6A-B. For example, as shown in FIGS. 6A and 6Brespective radiopaque alphanumeric characters are disposed at respectivelocations. For some applications, markers placed at respective locationsare identifiable based upon other features, e.g., based upon thedispositions of the markers relative to other markers. Using aradiographic imaging device (e.g., C-arm 34), a plurality ofradiographic images of the set of radiopaque markers are acquired,respective images being of respective locations along at least a portionof the subject's spine and each of the images including at least some ofthe radiopaque markers. Using computer processor 22, locations of theradiopaque markers within the radiographic images are identified, bymeans of image processing. At least some of the radiographic images arecombined with respect to one another based upon the identified locationsof the radiopaque markers within the radiographic images. Typically,such combination of images is similar to stitching of images. However,the images are typically not precisely stitched such as to stitchportions of the subject's anatomy in adjacent images to one another.Rather, the images are combined with sufficient accuracy to be able todetermine a location of the given vertebra within the combinedradiographic images.

For some applications, based upon the combined radiographic images, thecomputer processor automatically determines a location of the givenvertebra (e.g., the previously-marked targeted vertebra) within thecombined radiographic images. For some applications, the computerprocessor automatically determines location of the given vertebra withinthe combined radiographic images by counting the number of vertebrae onsaid image from an identifiable anatomical reference (e.g., the sacrum).For some applications, the counting is performed until theaforementioned N. For some applications, the counting is performed untila value that is defined relative to the aforementioned N. For someapplications, the vertebra(e) are counted automatically usingimage-processing techniques. For example, the image-processingtechniques may include shape recognition of anatomical features (ofvertebrae as a whole, of traverse processes, and/or of spinousprocesses, etc.). Or, the image-processing techniques may include outeredge line detection of spine (in a 2D image of the spine) and thencounting the number of bulges along the spine (each bulge correspondingto a vertebra). For some applications, the image-processing techniquesinclude techniques described in US 2010-0161022 to Tolkowsky, which isincorporated herein by reference. For some applications, the computerprocessor facilitates manual determination of the location of the givenvertebra within the combined radiographic images by displaying thecombined radiographic images.

Based upon the location of the given vertebra within the combinedradiographic images, a location of the given vertebra in relation to theset of radiopaque markers that is placed on or near the subject isdetermined, as described in further detail hereinbelow.

For some applications, a similar technique to that described hereinabovewith respect to vertebrae is performed with respect to a subject's ribs.For example, a set of radiopaque markers is placed upon or near thesubject, such that markers that are placed at respective locations withrespect to the subject are identifiable. Using a radiographic imagingdevice (e.g., C-arm 34), a plurality of radiographic images of the setof radiopaque markers are acquired, respective images being ofrespective locations along at least a portion of the subject's ribcageand each of the images including at least some of the radiopaquemarkers. Using the computer processor, locations of the radiopaquemarkers within the radiographic images are identified, by means of imageprocessing, and at least some of the radiographic images are combinedwith respect to one another based upon the identified locations of theradiopaque markers within the radiographic images.

For some applications, based upon the combined radiographic image, thecomputer processor automatically determines a location of the given rib(e.g., a targeted rib previously marked and counted within CT imagedata, MRI image data, and/or an x-ray image with a sufficiently largefield of view) within the combined radiographic images. For someapplications, the computer processor automatically determines locationof the given rib within the combined radiographic images by counting thenumber of ribs on the image from an identifiable anatomical reference.For some applications, the ribs are counted automatically usingimage-processing techniques. For example, the image-processingtechniques may include shape recognition of ribs. For some applications,the image-processing techniques include techniques described in US2010-0161022 to Tolkowsky, which is incorporated herein by reference.For some applications, the computer processor facilitates manualdetermination of the location of the given rib within the combinedradiographic images by displaying the combined radiographic images.Based upon the location of the given rib within the combinedradiographic images, a location of the given rib is determined inrelation to the set of radiopaque markers that is placed on or near thesubject.

It is noted that in the absence of sets 50 of markers 52, the typicalmethodology for determining the location of a given vertebra includesacquiring a series of x-rays along the patient's spine from the sacrum,and sticking radiopaque needles into the subject in order to match thex-rays to one another. Typically, in each x-ray spinal image only 3-4vertebrae are within the field of view, and multiple, overlapping imagesmust be acquired, such as to enable human counting of vertebra using theoverlapping images. This technique may involve switching back and forthbetween AP and lateral x-ray images. This method sometimes results inwrong-level surgery, as described, for example, in “Wrong-Site SpineSurgery: An Underreported Problem? AAOS Now,” American Association ofOrthopedic Surgeons, March 2010.

Reference is now made to FIG. 8B which shows an example of a 3D CT image95 of a subject's spine displayed alongside a combined radiographicimage 96 of the subject's spine, in accordance with some applications ofthe present invention. In accordance with some applications of thepresent invention, set 50 of markers 52 is placed upon or near thesubject, such that the bottom of the set of markers is disposed over, orin the vicinity of, the sacrum. A sequence of x-ray images fromgenerally the same view as one another are acquired along the spine,typically, but not necessarily, with some overlap between adjacentimages. The sequence of x-ray images is typically acquired from agenerally-AP view, but may also be acquired from a different view, suchas a generally-lateral view. Using computer processor 22, locations ofthe radiopaque markers within the radiographic images are identified, bymeans of image processing. At least some of the radiographic images arecombined with respect to one another based upon the identified locationsof the radiopaque markers within the radiographic images. For example,combined radiographic image 96 is generated by combining (a) a firstx-ray image 97 acquired from a generally-AP view and which starts at thesubject's sacrum and which includes markers H-E of the right marker setand markers 8-5 of the left marker set with (b) second x-ray image 98acquired from a generally similar view to the first view (but one whichis not exactly the same) and which includes markers E-B of the rightmarker set and markers 5-2 of the left marker set.

(It is noted that in FIG. 8B the alphanumeric markers appear as white inthe image. In general, the markers may appear as generally white orgenerally black, depending on (a) the contrast settings of the image(e.g., do radiopaque portions appear as white on a black background, orvice versa), and (b) whether the markers are themselves radiopaque, orthe markers constitute cut-outs from a radiopaque backing material, asis the case, in accordance with some applications of the presentinvention.)

Typically, the combination of images is similar to stitching of images.However, the images are typically not precisely stitched such as tostitch portions of the subject's anatomy in adjacent images to oneanother. Rather, the images are combined with sufficient accuracy tofacilitate counting vertebrae along the spine within the combined image.The physical location of a given vertebra is then known by virtue of itbeing adjacent to, or in the vicinity of, or observable in the x-rayimages relative to, a given one of the identifiable markers. It is notedthat in order to combine the radiographic images to one another, thereis typically no need to acquire each of the images from an exact view(e.g., an exact AP or an exact lateral view), or for there to be exactreplication of a given reference point among consecutive images. Rather,generally maintaining a given imaging direction, and having at leastsome of the markers generally visible in the images is typicallysufficient.

As described hereinabove, for some applications, the computer processorautomatically counts (and, for some applications, labels, e.g.,anatomically labels, and/or numerically labels) vertebrae within thecombined radiographic images in order to determine the location of thepreviously-marked target vertebra(e), or other vertebra(e) relative tothe previously marked vertebra. Alternatively, the computer processordrives the display to display the combined radiographic images such asto facilitate determination of the location of the previously-markedtarget vertebra(e) by an operator. The operator is able to count to thevertebra within the combined radiographic images, to determine, withinthe combined images, which of the radiopaque markers are adjacent to orin the vicinity of the vertebra, and to then physically locate thevertebra within the subject by locating the corresponding physicalmarkers.

Reference is now made to FIG. 8C, which shows an example of a 3D CTimage 100 of the subject's spine displayed alongside a 2D radiographicimage 102 of the subject's spine, in accordance with some applicationsof the present invention. As shown, markers 52 appear on the combinedradiographic image. As shown, vertebra 91, which was identified by anoperator with respect to the 3D image data (as described hereinabovewith reference to FIG. 8A), has been identified within the 2Dradiographic image using the above-described techniques, and is denotedby cursor 104.

For some applications, based upon counting and/or labeling of thevertebrae in the combined radiographic image, computer processor 22 ofsystem 20 counts and/or labels vertebrae within the 3D image data (e.g.,a 3D image, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data). For some applications, thecomputer processor drives the display to display the labeled vertebraewhile respective corresponding 2D images are being acquired anddisplayed. Alternatively or additionally, the computer processor drivesthe display to display the labeled vertebrae when the combinedradiographic image has finished being generated and/or displayed. It isnoted that, typically, the computer processor counts, labels, and/oridentifies vertebrae on the 3D image data and on the 2D radiographicimages without needing to determine relative scales of the 3D image dataand 2D images. Rather, it is sufficient for the computer processor to beable to identify individual vertebrae at a level that is sufficient toperform the counting, labeling, and/or identification of vertebrae.

It is noted that the above-described identification of vertebrae that isfacilitated by markers 52 is not limited to being performed by thecomputer processor at the start of an intervention. Rather, the computerprocessor may perform similar steps at subsequent stages of theprocedure. Typically, it is not necessary for the computer processor torepeat the whole series of steps at the subsequent stages, since thecomputer processor utilizes knowledge of an already-identified vertebra,in order to identify additional vertebrae. For example, afteridentifying and then performing a procedure with respect to a firstvertebra, the computer processor may utilize the combined radiographicimage to derive a location of a further target vertebra (which may beseparated from the first vertebra by a gap), based upon thealready-identified first vertebra. For some applications, in order toderive the location of a further target vertebra, the computer processorfirst extends the combined radiographic image (typically, using themarkers in order to do so, in accordance with the techniques describedhereinabove).

For some applications, the operator labels a single vertebra (or in someapplications, a plurality of vertebra) within the 3D image data of thespine. Based upon the labelling of the vertebra(e) the computerprocessor automatically labels other vertebrae within the 3D image data,based upon the known anatomical sequence of vertebrae along the spine.For some applications, based upon the labelling of the one or morevertebrae within the 3D image data, the computer processor labels (e.g.,anatomically labels, and/or numerically labels) the vertebrae within thecombined radiographic image. In this manner, a spinal roadmap is createdwithin the 3D image data (e.g., a 3D image, a 2D cross-section derivedfrom 3D image data, and/or a 2D projection image derived from 3D imagedata) and within the combined radiographic images. For someapplications, the spinal roadmap is generated upon an image that is afused combination of the 3D image data and the 2D radiographic images.For some applications, the spinal roadmap is automatically updatedintraprocedurally. For example, in response to detecting a tool within agiven vertebra within a 2D x-ray image, the 3D spinal roadmap may beupdated to show the tool within the vertebra, using the coregistrationtechniques described in detail hereinbelow.

In general, the scope of the present invention includes acquiringimaging data of the subject's spine, using an imaging device (e.g., a 2Dimaging device, and/or a 3D imaging device). During a procedure in whichinterventions are performed with respect to at least first and secondvertebrae of a subject's spine, a spinal roadmap image of at least aportion of the spine that contains the first and second vertebra isgenerated by computer processor 22 and displayed upon display 30. Thecomputer processor typically automatically labels vertebra within thespinal roadmap image. For some applications, the computer processordetermines that an intervention has been performed with respect to thefirst vertebra (e.g., a tool has been inserted and/or implanted in thefirst vertebra), such that an appearance of the first vertebra haschanged. In response thereto, the computer processor automaticallyupdates the spinal roadmap to reflect the change in the appearance ofthe first vertebra, such that the updated spinal roadmap is displayedwhile the intervention is performed with respect to the second vertebra.

Reference is now made to FIG. 9, which shows an example of an opticalimage 110 displayed alongside a 2D radiographic (e.g., x-ray) image 112,in accordance with some applications of the present invention. Asdescribed with reference to step 76 of FIG. 7, subsequent to identifyinga target vertebra along the subject's spine, typically, the operatordetermines a desired site for an incision (in the case ofminimally-invasive surgery) or tool insertion (in the case of opensurgery). For some applications, in order to facilitate thedetermination of the incision site or tool insertion site, an opticalcamera 114 is disposed within the operating room such that the opticalcamera has a generally similar viewing angle to that of the 2Dradiographic imaging device. For example, the camera may be disposed onx-ray C-arm 34, as shown in FIG. 1. Alternatively or additionally, thecamera may be disposed on a separate arm, may be handheld, may be theback camera of a display such as a tablet or mini-tablet device, and/ormay be held by another member of the operating room staff. For someapplications, the camera is placed on the surgeon's head. Typically, forsuch applications, the surgeon uses a head-mounted display.

For some applications, a 2D radiographic image 112 of a portion of thesubject's body is acquired in a radiographic imaging modality, using the2D radiographic imaging device (e.g., C-arm 34), and an optical image110 of the subject's body is acquired in optical imaging modality, usingoptical camera 114 (shown in FIG. 1). Computer processor 22 of system 20identifies radiopaque markers (e.g., markers 52) in the radiographicimage and in the optical image, by means of image processing. By way ofexample, in FIG. 9, radiopaque gridlines (as shown in FIG. 5B), andalphanumeric radiopaque markers associated with the radiopaque gridlines(also as shown in FIG. 5B), are visible in both the radiographic and theoptical image. Based upon the identification of the radiopaque markersin the radiographic image and in the optical image, the computerprocessor bidirectionally maps the radiographic image and the opticalimage with respect to one another. It is noted that acquisition of theradiographic image and the optical image from generally-similar views(but not necessarily identical views) is typically sufficient tofacilitate the bidirectional mapping of the images to one another, byvirtue of the radiopaque markers that are visible in both of the images.

As shown in FIG. 9, for some applications, the computer processor drivesdisplay 30 to display the radiographic image and the optical imageseparately from one another, upon one or more displays. Subsequently, inresponse to receiving an input indicating a location in a first one ofthe radiographic and the optical images, the computer processorgenerates an output indicating the location in the other one of theradiographic and the optical images. For example, in response to a lineor a point being marked on 2D x-ray image 112, the computer processorindicates a corresponding lines or points overlaid on the optical image110. Similarly, in response to a line or a point being marked on opticalimage 110, the computer processor indicates a corresponding lines orpoints overlaid on the 2D x-ray image 112. Further similarly, inresponse to a line or a point being marked on, or an object such as ak-wire or incision knife laid upon, the subject's body (e.g., back inthe case of a planned dorsal tool insertion) as seen in a then-currentoptical image 110, the computer processor identifies such line, point orobject (or applicable portion thereof) and indicates a correspondinglines or points overlaid on the 2D x-ray image 112. For someapplications, a line or point is drawn on the subject's body (e.g., onthe subject's back in the case of a planned dorsal tool insertion) usingradiopaque ink.

Traditionally, in order to determine the location of an incision site, aradiopaque wire is placed on the subject's back at a series oflocations, and the x-rays are taken of the wire at the locations, untilthe incision site is determined. Subsequently, a knife is placed at thedetermined incision site, and a final x-ray image is acquired forverification. By contrast, in accordance with the technique describedherein, initially a single x-ray image may be acquired andbidirectionally mapped to the optical image. Subsequently the wire isplaced at a location, and the corresponding location of the wire withrespect to the x-ray image can be observed (using the bidirectionalmapping) without requiring the acquisition of a new x-ray image.Similarly, when an incision knife is placed at a location, thecorresponding location of an applicable portion of the knife (typically,its distal tip) with respect to the x-ray image can be observed (usingthe bidirectional mapping) without requiring the acquisition of a newx-ray image. Alternatively or additionally, a line can be drawn on thex-ray image (e.g., a vertical line that passes along the vertebralcenters, anatomically along the spinous processes of the vertebrae) andthe corresponding line can be observed in the optical image overlaid onthe patient's back.

For some applications, a surgeon places a radiopaque knife 116 (oranother radiopaque tool or object) at a prospective incision site(and/or places a tool at a prospective tool insertion location) andverifies the location of the incision site (and/or tool insertionlocation) by observing the location of the tip of the knife (or portionof another tool) with respect to the x-ray (e.g., via cursor 117), bymeans of the bi-directional mapping between the optical image and thex-ray image. For some applications, the functionalities describedhereinabove with reference to FIG. 9, and/or with reference otherfigures, are performed using markers (which are typically sterile),other than markers 52. For example, a radiopaque shaft 118, ruler,radiopaque notches, and/or radiopaque ink may be used.

Reference is now made to FIG. 10, which shows an example of a 2Dradiographic (e.g., x-ray) image 120 displayed alongside across-sectional image 122 of a subject's vertebra that is derived from a3D image data of the vertebra, in accordance with some applications ofthe present invention. For some applications, even prior to registeringthe 2D images to the 3D image data (as described hereinbelow), thefollowing steps are performed. X-ray image 120 of a given view thesubject's spine (e.g., AP, as shown) is acquired. A point is indicatedupon the image, e.g., the point indicated by cursor 124 in FIG. 10.Computer processor 22 of system 20 automatically identifies the endplates of the vertebra, and calculates the relative distance ofindicated point from end plates. (It is noted that the computerprocessor typically does not calculate absolute distances in order toperform this function.) From the 3D (e.g., CT) image of the samevertebra, the computer processor generates and displays a cross-sectionof a given plane (which is typically axial) at the indicated location(e.g. image 122). For some applications, upon the cross-section, thecomputer processor drives the display to show a line 126 (e.g., avertical line) within the cross-section, the line indicating that theindicated location falls somewhere along the line. For someapplications, the line is drawn vertically upon an axial cross-sectionof the vertebra as shown. The computer processor determines where toplace the line according to distance of the indicated point from leftand right edges of the vertebra, and/or according to the position of theindicated point relative to visible features (e.g., spinous process,traverse processes, pedicles) in the x-ray image. Typically, thecross-sectional image with the line, and coupled with the surgeon'stactile feel of how far from the vertebra the skin is (and/or derivingsuch information from a 3D image), assists the surgeon in calculatingthe desired insertion angle of a tool.

Referring again to step 78 of FIG. 7, the first tool in the sequence oftools (which is typically a needle, e.g., a Jamshidi™ needle) isinserted into the subject (e.g., in the subject's back), and is slightlyfixated in the vertebra. Subsequently, in step 80, two or more 2Dradiographic images are acquired from respective views that typicallydiffer by at least 10 degrees (e.g., 30 degrees or more), and one ofwhich is typically from the direction of insertion of the tool. Commoncombinations of such views include AP and left or right lateral, AP withleft or right oblique, left oblique with left lateral, and right obliquewith right lateral. It is noted that, as described in further detailhereinbelow, with reference to FIG. 15A, for some applications, 2Dradiographic images of the tool and the vertebra are acquired from onlya single x-ray image view.

Reference is now made to FIGS. 11A and 11B, which show examples ofrespectively AP and lateral x-ray images of a Jamshidi™ needle 36 beinginserted into a subject's spine, in accordance with some applications ofthe present invention. As shown, sets 50 of markers 52 typically appearat least in the AP image.

Reference is now made to FIGS. 12A and 12B, which show examples ofcorrespondence between views of a 3D image of a vertebra, with,respectively, first and second 2D x-ray images 132 and 136 of thevertebra, in accordance with some applications of the present invention.In FIG. 12A the correspondence between a first view 130 of a 3D image ofa vertebra with an AP x-ray image of the vertebra is shown, and in FIG.12B the correspondence between a second view 134 of a 3D image of avertebra with a lateral x-ray image of the vertebra is shown.

For some applications, subsequent to the fixation of the tool in thesubject's vertebra, the 3D image data and 2D images are registered toeach other, in accordance with step 82 of FIG. 7. However, it is notedthat the registration of the 3D image data and the 2D images to eachother may be performed even in the absence of a tool within the images,in accordance with the techniques described hereinbelow. Typically, the3D image data and 2D images are registered to each other by generating aplurality of 2D projections from the 3D image data, and identifyingrespective first and second 2D projections that match the first andsecond 2D x-ray images of the vertebra, as described in further detailhereinbelow. (For some applications, 2D x-ray images from more than two2D x-ray image views are acquired and the 3D image data and 2D x-rayimages are registered to each other by identifying a correspondingnumber of 2D projections of the 3D image data that match respective 2Dx-ray images.) Typically, the first and second 2D x-ray images of thevertebra are acquired using an x-ray imaging device that is unregisteredwith respect to the subject's body, by (a) acquiring a first 2D x-rayimage of the vertebra (and at least a portion of the tool) from a firstview, while the x-ray imaging device is disposed at a first pose withrespect to the subject's body, (b) moving the x-ray imaging device to asecond pose with respect to the subject's body, and (c) while the x-rayimaging device is at the second pose, acquiring a second 2D x-ray imageof at least the portion of the tool and the skeletal portion from asecond view.

For some applications, the 3D imaging that is used is CT imaging, andthe following explanation of the registration of the 3D image data tothe 2D images will focus on CT images. However, the scope of the presentinvention includes applying the techniques describe herein to other 3Dimaging modalities, such as MRI and 3D x-ray, mutatis mutandis.

X-ray imaging and CT imaging both apply ionizing radiation to image anobject such as a body portion or organ. 2D x-ray imaging generates aprojection image of the imaged object, while a CT scan makes use ofcomputer-processed combinations of many x-ray images taken fromdifferent angles to produce cross-sectional images (virtual “slices”) ofthe scanned object, allowing the user to see inside the object withoutcutting. Digital geometry is used to generate a 3D image of the insideof the object from a large series of 2D images.

Reference is now made to FIGS. 13A, 13B, and 13C, which demonstrate therelationship between a 3D image of an object (which in the example shownin FIG. 13A is a cone) and side-to-side (FIG. 13B) and bottom-to-top(FIG. 13C) 2D images of the object, such relationship being utilized, inaccordance with some applications of the present invention. As shown,for the example of the cone, the bottom-to-top 2D image (which isanalogous to an AP x-ray image of an object acquired by C-arm 34, asschematically indicated in FIG. 13C) is a circle, while the side-to-sideimage (which is analogous to a lateral x-ray image of an object,acquired by C-arm 34, as schematically indicated in FIG. 13C) is atriangle. It follows that, in the example shown, if the circle and thetriangle can be registered in 3D space to the cone, then they alsobecome registered to one another in that 3D space. Therefore, for someapplications, 2D x-ray images of a vertebra from respective views areregistered to one another and to 3D image data of the vertebra bygenerating a plurality of 2D projections from the 3D image data, andidentifying respective first and second 2D projections that match the 2Dx-ray images of the vertebra.

In the case of 3D CT images, the derived 2D projections are known asDigitally Reconstructed Radiographs (DRRs). If one considers 3D CT dataand a 2D x-ray image of the same vertebra, then a simulated x-ray cameraposition (i.e., viewing angle and viewing distance) can be virtuallypositioned anywhere in space relative to a 3D image of the vertebra, andthe corresponding DRR that this simulated camera view would generate canbe determined. At a given simulated x-ray camera position relative tothe 3D image of the vertebra, the corresponding DRR that this simulatedcamera view would generate is the same as the 2D x-ray image. For thepurposes of the present application, such a DRR is said to match anx-ray image of the vertebra. Typically, 2D x-ray images of a vertebrafrom respective views are registered to one another and to 3D image dataof the vertebra by generating a plurality of DRRs from 3D CT image data,and identifying respective first and second DRRs (i.e., 2D projections)that match the 2D x-ray images of the vertebra. By identifyingrespective DRRs that match two or more x-ray images acquired fromrespective views, the x-ray images are registered to the 3D image data,and, in turn, the x-ray images are registered to one another via theirregistration to the 3D image data.

For some applications, in order to register the 2D images to the 3Dimage data, additional registration techniques are used in combinationwith the techniques described herein. For example, intensity basedmethods, feature based methods, similarity measures, transformations,spatial domains, frequency domains, etc., may be used to perform theregistration.

Typically, by registering the x-ray images to the 3D image data usingthe above-described technique, the 3D image data and 2D x-ray images arebrought into a common reference frame to which they are all aligned andscaled. It is noted that the registration does not require tracking thesubject's body or a portion thereof (e.g., by fixating one or morelocation sensors, such as an IR light, an IR reflector, an opticalsensor, or a magnetic or electromagnetic sensor, to the body or bodyportion, and tracking the location sensors).

Typically, between preprocedural 3D imaging (e.g., 3D imaging performedprior to entering the operating room, or prior to performing a givenintervention) and intraprocedural 2D imaging, the position and/ororientation of a vertebra relative to the subject's body and toneighboring vertebrae is likely to change. For example, this may be dueto the patient lying on his/her back in preprocedural imaging but on thestomach or on the side for intraprocedural imaging, or the patient'sback being straight in preprocedural imaging, but being folded (e.g., ona Wilson frame) in intraprocedural imaging. In addition, in some cases,due to anesthesia the position of the spine changes (e.g. sinks), andonce tools are inserted into a vertebra, that may also change itspositioning relative to neighboring vertebrae. However, since a vertebrais a piece of bone, its shape typically does not change between thepreprocedural 3D imaging and the intraprocedural 2D imaging. Therefore,registration of the 3D image data to the 2D images is typicallyperformed with respect to individual vertebrae. For some applications,registration of the 3D image data to the 2D images is performed on aper-vertebra basis even in cases in which segmentation of a vertebra inthe 3D image data leaves some elements, such as portions of the spinousprocesses of neighboring vertebrae, within the segmented image of thevertebra. In addition, for some applications, registration of the 3Dimage data to the 2D images is performed with respect to a spinalsegment comprising several vertebrae. For example, registration of 3Dimage data to the 2D images may be performed with respect to a spinalsegment in cases in which the 3D image data were acquired when thesubject was already in the operating room and positioned upon thesurgical table for the intervention.

As described hereinabove, typically, during a planning stage, anoperator indicates a target vertebra within the 3D image data of thespine or a portion thereof (e.g., as described hereinabove withreference to FIG. 8A). For some applications, the computer processorautomatically identifies the target vertebra in the x-ray images, bymeans of image processing, e.g., using the techniques describedhereinabove. For some applications, the registration of the 3D imagedata to the 2D images is performed with respect to an individualvertebra that is automatically identified, by the computer processor, ascorresponding to a target vertebra as indicated by the operator withrespect to the 3D image data of the spine or a portion thereof (e.g., asdescribed hereinabove with reference to FIGS. 8A-C).

Typically, and since the registration is performed with respect to anindividual vertebra, the registration is not affected by motion of thevertebra that occurs between the acquisition of the two x-ray images(e.g., due to movement of the subject upon the surgical table, motiondue to respiration, etc.), since both motion of the C-arm and of thevertebra may be assumed to be rigid transformations (and thus, if bothmotions occur in between the acquisition of the two x-ray images, achaining of two rigid transformations may be assumed).

For some applications, motion of the patient is detected in order toserve as an input for some functionalities of computer processor 22. Forexample, a motion detection sensor 56 may be coupled to a set 50 ofmarkers 52 (e.g., by being coupled to drape 53, as shown, by way ofexample, in FIG. 5A), and/or to a portion of the subject's body. It isnoted that the motion detection sensor is typically not a locationsensor, the motion of which is tracked by a tracker, or a tracker thatis configured to track the motion of a location sensor situatedelsewhere. Typically, the motion detection sensor detects that a portionof the subject's body has undergone motion in a standalone manner (i.e.,the motion detection sensor detects that its motion has occurredrelative to a prior position of itself (as opposed to detecting that itsmotion has occurred relative to an external element)). Furthermore,typically, the motion detection sensor is configured to detect thatmotion has occurred, but not necessarily that a specific motion hasoccurred. For example, the motion detection sensor may be configured todetect motion that is greater than a threshold. For some applications,an accelerometer is used. For example, the accelerometer could beconfigured to detect motion that is abrupt. In response to detectingthat motion has occurred, the motion detection sensor may communicate asignal to the computer processor. For some applications, in response toreceiving an input indicating that the subject has undergone motion, thecomputer processor generates an alert, and/or generates an outputindicating that one or more images should be re-acquired. Alternativelyor additionally, the motion detection sensor is independently poweredand is configured to generate an alert, e.g., a visual or an audio alert(e.g., via a light or buzzer attached thereto), in response to detectingthat motion has occurred.

In general, the scope of the present invention includes acquiring asequence of two or more images of a subject's body, in order todetermine the location of a tool with respect to the body, during amedical intervention. For some applications, during such a procedure, amotion detection sensor is configured to detect that motion of thesubject (or a portion of the subject) that is greater than a giventhreshold has occurred, for example, the motion detection sensor may beconfigured to detect that such motion has occurred between theacquisitions of two or more of the images. Typically, the motiondetection sensor detects that a portion of the subject's body hasundergone motion in a standalone manner (i.e., the motion detectionsensor detects that its motion has occurred relative to a prior positionof itself (as opposed to detecting that its motion has occurred relativeto an external element)). In response thereto, the motion sensorgenerates an alert indicating to a user that such motion has occurred.For some applications, the motion detection sensor generates an outputby driving an output device itself. Alternatively or additionally, inresponse to receiving an input from a motion detection sensor indicatingthat such motion has occurred, a computer processor (e.g., computerprocessor 22) generates an alert indicating to a user that such motionhas occurred. For some applications, the computer processor generates anoutput advising the user to acquire additional images (e.g., toreacquire an image from a given imaging view).

As described hereinabove, typically, 2D x-ray images of a vertebra fromrespective views are registered to one another and to a 3D image data ofthe vertebra by generating a plurality of DRRs from a 3D CT image, andidentifying respective first and second DRRs that match the 2D x-rayimages of the vertebra. By identifying respective DRRs that match two ormore x-ray images acquired from respective views, the x-ray images areregistered to the 3D image data, and, in turn, the x-ray images areregistered to one another via their registration to the 3D image data.

For some applications, in order to avoid double solutions when searchingfor a DRR that matches a given x-ray image, computer processor 22 firstdetermines whether the x-ray image is, for example, AP, PA, leftlateral, right lateral, left oblique, or right oblique, and/or fromwhich quadrant a tool is being inserted. The computer processor maydetermine this automatically, e.g., by means of sets 50 of markers 52,using techniques described herein. Alternatively, such information maybe manually inputted into the computer processor.

For some applications, in order to identify a DRR that matches a givenx-ray image, computer processor 22 first limits the search space withinwhich to search for a matching DRR, e.g., by using techniques such asthose described in U.S. Pat. No. 9,240,046 to Carrell, which isincorporated herein by reference.

For some applications, the steps of generating a plurality of DRRs froma 3D CT image, and identifying respective first and second DRRs thatmatch the 2D x-ray images of the vertebra are aided by deep-learningalgorithms. In general, for such applications, during a learning stage,many sets of 3D CT images of vertebra and x-ray images of those samevertebra are inputted into a computer processor which functions as adeep learning engine. The registered outcome for each set, i.e., theDRRs that match the x-rays, are determined. The results of the deeplearning are then inputted to computer processor 22. Subsequently,intraprocedurally, computer processor 22 uses the results of the deeplearning stage to facilitate the matching of DRRs from the CT image ofthe subject's vertebra to x-ray images.

For some applications, deep-learning techniques are performed as part ofthe processing of images of a subject's vertebra, as described in thefollowing paragraphs. By performing the deep-learning techniques, thesearch space for DRRs of the subject's vertebra that match the x-rayimages is limited, which reduces the intraprocedural processingrequirement, reduces the time taken to performing the matching, and/orreduces cases of dual solutions to the matching.

For some applications, in a first deep-learning phase, a moderate number(e.g., fewer than 10,000, or fewer than 1000, which is moderate relativeto much larger data sets that are typically required for deep learning)spinal CT scans are processed, each of the CT scans comprising multiplevertebrae, for example, above 20. For each vertebra, a large number ofpairs (or triplets, or greater multiples) of DRRs are generated, eachpair being generated from simulated viewing distances and simulatedcamera angles that are typically at least 10 degrees apart. Thesimulated camera angles are those that are typically used in x-rayacquisition during spinal surgery, such as generally-AP,generally-left-oblique, generally-right-oblique, generally-left-lateral,and/or generally-right-lateral. All of these sets, each comprising,typically, a 3D CT and a DRR pair and the simulated camera viewingdistances and angles from which the DRRs were generated, are fed into adeep-learning analytical engine. Thus, the engine learns, given avertebral 3D CT and a pair of DRRs, to suggest simulated camera viewingdistances and angles that correspond to those DRRs. Subsequently, thedeep-learning data is fed as an input to computer processor 22 of system20. Intraprocedurally, in order to register the 2D x-ray images to the3D image data, computer processor uses the deep-learning data in orderto limit the search space in which DRRs of the 3D image data that matchthe x-ray images should be searched for. Computer processor 22 thensearches for the matching DRRs only within the search space that wasprescribed by the deep-learning data.

Alternatively or additionally, during the deep-learning phase, a largedatabase of 2D x-ray images and (at least some of) their knownparameters relative to vertebra are inputted to a deep-learning engine.Such parameters typically include viewing angle, viewing distance, andoptionally additional camera parameters. For some applications, theaforementioned parameters are exact. Alternatively, the parameters areapproximate parameters. The parameters may be recorded originally whengenerating the images, or annotated by a radiologist. Thus, the enginelearns, given a certain 2D projection image, to suggest simulated cameraviewing distances and angles that correspond to that projection image.Subsequently, the deep-learning data is fed as an input to computerprocessor 22 of system 20. Intraprocedurally, in order to register the2D x-ray images to the 3D image data, computer processor uses thedeep-learning data in order to limit the search space in which DRRs ofthe 3D image data that match the x-ray images should be searched for.Computer processor 22 then searches for the matching DRRs only withinthe search space that was prescribed by the deep-learning data.

The above-described registration steps are summarized in FIG. 14A, whichis a flowchart showing steps that are performed by computer processor,in order to register 3D image data of a vertebra to two or more 2D x-rayimages of the vertebra.

In a first step 140, the search space for DRRs that match respectivex-ray images is limited, for example, using deep-learning data, and/orusing techniques such as those described in U.S. Pat. No. 9,240,046 toCarrell, which is incorporated herein by reference. Alternatively oradditionally, in order to avoid double solutions when searching for aDRR that matches a given x-ray image, the computer processor determineswhether the x-ray images are, for example, AP, PA, left lateral, rightlateral, left oblique, or right oblique, and/or from which quadrant atool is being inserted.

In step 141, a plurality of DRRs are generated within the search space.

In step 142, the plurality of DRRs are compared with the x-ray imagesfrom respective views of the vertebra.

In step 143, based upon the comparison, the DRR that best matches eachof the x-ray images of the vertebra is selected. Typically, for thesimulated camera position that would generate the best-matching DRR, thecomputer processor determines the viewing angle and viewing distance ofthe camera from the 3D image of the vertebra.

It is noted that the above steps are performed separately for each ofthe 2D x-ray images that is used for the registration. For someapplications, each time one or more new 2D x-ray images are acquired,the image(s) are automatically registered to the 3D image data using theabove described technique. The 2D to 3D registration is thereby updatedbased upon the new 2D x-ray acquisition(s).

Reference is now made to FIG. 14B, which is a flowchart showing steps ofan algorithm that is performed by computer processor 22 of system 20, inaccordance with some applications of the present invention.

As described hereinabove, for each of the x-ray images (denoted X1 andX2), the computer processor determines a corresponding DRR from asimulated camera view (the simulated cameras being denoted C1 for X1 andC2 for X2).

The 3D scan and two 2D images are now co-registered, and the following3D-2D bi-directional relationship generally exists:

Geometrically, a point P3D in the 3D scan of the body portion (in threecoordinates) is at the intersection in 3D space of two straight lines

-   -   i. A line drawn from simulated camera C1 through the        corresponding point PX1 (in two image coordinates) in 2D image        X1.    -   ii. A line drawn from simulated camera C2 through the        corresponding point PX2 (in two image coordinates) in 2D image        X2.

Therefore, referring FIG. 14B, for some applications, for a portion of atool that is visible in the 2D images, such as the tool tip or a distalportion of the tool, the computer processor determines its locationwithin the 3D image data (denoted TP3D), using the following algorithmicsteps:

Step 145: Identify, by means of image processing, the tool's tip TPX1 inimage X1 (e.g., using the image processing techniques describedhereinabove). For some applications, to make the tool tip point betterdefined, the computer processor first generates a centerline for thetool and then the tool's distal tip TPX1 is located upon on thatcenterline.

In general, the computer processor identifies the locations of a tool ora portion thereof in the 2D x-ray images, typically, solely by means ofimage processing. For example, the computer processor may identify thetool by using a filter that detects pixel darkness (the tool typicallybeing dark), using a filter that detects a given shape (e.g., anelongated shape), and/or by using masks. For some applications, thecomputer processor compares a given region within the image to the sameregion within a prior image. In response to detecting a change in somepixels within the region, the computer processor identifies these pixelsas corresponding to a portion of the tool. For some applications, theaforementioned comparison is performed with respect to a region ofinterest in which the tool is likely to be inserted, which may be basedupon a known approach direction of the tool. For some applications, thecomputer processor identifies the portion of the tool in the 2D images,solely by means of image processing, using algorithmic steps asdescribed in US 2010-0161022 to Tolkowsky, which is incorporated hereinby reference.

Step 146: Generate a typically-straight line L1 from C1 to TPX1. (It isnoted that, as with other steps described as being performed by thecomputer processor, the generation of a line refers to a processing stepthat is the equivalent of drawing a line, and should not be construed asimplying that a physical line is drawn. Rather the line is generated asa processing step).Step 147: Identify, by means of image processing, the tool's tip TPX2 inimage X2 (e.g., using the image processing techniques describedhereinabove). For some applications, to make the tool tip point betterdefined, the computer processor first generates a centerline for thetool and then the tool's distal tip TPX2 is located upon on thatcenterline. The image processing techniques that are used to tool's tipTPX2 in image X2 are generally similar to those described above withreference to step 145.Step 148: Generate a typically-straight line L2 from C2 to TPX2.Step 149: Identify the intersection of L1 and L2 in 3D space as thelocation of the tool's tip relative to the 3D scan data.Step 150: Assuming that the shape of the tool is known (e.g., if thetool is a rigid or at least partially rigid tool, or if the tool can beassumed to have a given shape by virtue of having been placed intotissue), the computer processor derives the locations of additionalportions of the tool within 3D space. For example, in case of a toolwith straight shaft in whole or in its distal portion, or one that maybe assumed to be straight once inserted into bone, or at least straightin its distal portion once inserted into bone, then this shaft, or atleast its distal portion, resides at the intersection of two planes,each extending from the simulated camera to the shaft (or portionthereof) in the corresponding 2D image. For some applications, thedirection of the shaft from its tip to proximal and along theintersection of the two planes is determined by selecting a pointproximally to the tool's tip on any of the x-ray images and observingwhere a line generated between such point and the correspondingsimulated camera intersects the line of intersection between the twoplanes.

It is noted that, since the coregistration of the 3D image data to the2D images is bidirectional, for some applications, the computerprocessor identifies features that are identifiable within the 3D imagedata, and determines the locations of such features with respect to the2D x-rays, as described in further detail hereinbelow. The locations ofeach such feature with respect to any of the 2D x-rays are typicallydetermined by (a) generating a typically-straight line from thesimulated camera that was used to generate the DRR corresponding to suchx-ray image and through the feature within the 3D image data and (b)thereby determining the locations of the feature with respect to thex-ray images themselves. For some applications, the locations of suchfeatures with respect to the 2D x-ray images are determined bydetermining the locations of the features within the DRRs that match therespective x-ray images, and assuming that the features will be atcorresponding locations within the matching x-ray images.

For some applications, based upon the registration, 3D image data isoverlaid upon a 2D image. However, typically, the 3D image data (e.g., a3D image, a 2D cross-section derived from 3D image data, and/or a 2Dprojection image derived from 3D image data) are displayed alongside 2Dimages, as described in further detail hereinbelow.

Reference is now made to FIG. 15A, which shows an example ofcross-sections 160 and 162 of a vertebra corresponding, respectively, tofirst and second locations of a tip 164 of a tool that is advanced intothe vertebra along a longitudinal insertion path, as shown oncorresponding 2D x-ray images, in accordance with some applications ofthe present invention. Typically, the tool has a straight shaft in wholeor in its distal portion, and/or may be assumed to be straight onceinserted into bone, or at least straight in its distal portion onceinserted into bone. Referring also to step 84 of FIG. 7, for someapplications, based upon the identified location of the tip of tool withrespect to one or more 2D x-ray image of the vertebra that are acquiredfrom a single image view, and the registration of an x-ray from thesingle 2D x-ray image view to the 3D image data (e.g., by matching a DRRfrom the 3D image data to the 2D x-ray image), computer processor 22determines a location of the tip of the tool with respect to a DRR thatis derived from the 3D image data (e.g., the DRR that was determined tomatch the 2D x-ray image), and in response thereto, drives the displayto display a cross-section of the vertebra, the cross-section beingderived from the 3D image data, and corresponding to the location of thetool tip. The cross-section is typically of a given plane at theidentified location. Typically, the cross-section is an axialcross-section, but for some applications, the cross-section is asagittal cross-section, a coronal cross-section, and/or a cross-sectionthat is perpendicular to or parallel with the direction of the toolinsertion.

For some applications, upon the cross-section, the computer processordrives the display to show a line 166 (e.g., a vertical line),indicating that the location of the tip of the tool is somewhere alongthat line. For some applications, the line is drawn vertically upon anaxial cross-section of the vertebra, as shown. For some applications,the surgeon is able to determine the likely location of the tool alongthe line based upon their tactile feel. Alternatively or additionally,based on the 3D image data, the computer processor drives the display todisplay how deep below the skin the vertebra is disposed, which acts asa further aid to the surgeon in determining the location of the toolalong the line.

As noted above, typically it is possible to generate an output as shownin FIG. 15A, by acquiring one or more 2D x-ray images from only a singlex-ray image view of the tool and the vertebra, and registering one ofthe 2D x-ray images to the 3D image data using the registrationtechniques described herein. Typically, by registering the 2D x-rayimage acquired from the single image view to the 3D image data, computerprocessor 22 determines, with respect to 3D image data (e.g., withrespect to the DRR that was determined to match the 2D x-ray image), (a)a plane in which the tip of the tool is disposed, and (b) a line withinthe plane, somewhere along which the tip of the tool is disposed, asshown in FIG. 15A. As described hereinabove, typically, when the tip ofthe tool is disposed at an additional location with respect to thevertebra, further 2D x-ray images of the tool at the additional locationare acquired from the same single x-ray image view, or a differentsingle x-ray image view, and the above-described steps are repeated.Typically, for each location of the tip of the tool to which theabove-described technique is applied, 2D x-ray images need only beacquired from a single x-ray image view, which may stay the same for therespective locations of the tip of the tool, or may differ forrespective locations of the tip of the tool.

Reference is now made to FIG. 15B, which is a schematic illustration ofthe location of the tool tip 168 denoted by cross-hairs uponcross-sections 160 and 162 of the vertebra corresponding, respectively,to first and second locations of a tip 164 of a tool that is advancedinto the vertebra along a longitudinal insertion path (as shown in FIG.15A), in accordance with some applications of the present invention. Forsome applications, by initially registering two or more 2D x-ray imagesof the tool and the vertebra that were acquired from respective 2D x-rayimage views, to the 3D image data, the precise location of the tip ofthe tool within a cross-section derived from the 3D image data isdetermined and indicated upon the cross-section, as shown in FIG. 15B.As described hereinbelow, with reference to FIGS. 19A-19B, for someapplications, after initially determining the location of the tip of thetool with respect to the 3D image data using two or more 2D x-ray imagesof the tool and the vertebra that were acquired from respective 2D x-rayimage views, subsequent locations of the tip of the tool are determinedwith respect to the 3D image data by acquiring further x-ray images fromonly a single x-ray image view.

Reference is now made to FIGS. 16A and 16B, which show examples of adisplay showing a relationship between an anticipated longitudinalinsertion path 170 of a tool 172 and a designated location 174 upon,respectively, AP and lateral 2D x-ray images, in accordance with someapplications of the present invention. Reference is also made to step 86of FIG. 7.

For some applications, a location within a vertebra is designated withinthe 3D image data. For example, an operator may designate a targetportion (e.g. a fracture, a tumor, a virtual pedicle screw, etc.),and/or a region which the tool should avoid (such as the spinal cord)upon the 3D image data (e.g., a 3D image, a 2D cross-section derivedfrom 3D image data, and/or a 2D projection image derived from 3D imagedata). Alternatively or additionally, the computer processor mayidentify such a location automatically, e.g., by identifying the portionvia image processing. Based upon the registration of the first andsecond 2D x-ray images to the 3D image data, the computer processorderives a position of the designated location within at least one of thex-ray images, using the techniques described hereinabove. In addition,the computer processor determines an anticipated path of the tool withinthe x-ray image. Typically, the computer processor determines theanticipated path by determining a direction of an elongate portion ofthe tool (and/or a center line of the elongate portion) within the x-rayimage. Since the tool is typically advanced along a longitudinalinsertion path, the computer processor extrapolates the anticipated pathby extrapolating a straight line along the determined direction.

For some applications, the computer processor performs a generallysimilar process, but with respect to a desired approach vector (e.g.,for insertion and implantation of a screw) that, for example, is inputinto the computer processor manually, and/or is automatically derived bythe processor. For example, such an approach vector may have beengenerated during a planning phase, typically upon the 3D image data, andbased upon the insertion of a simulated tool into the vertebra.Typically, such an approach vector is one that reaches a desired target,while avoiding the spinal cord or exiting the vertebra sideways.

For some applications, in response to the above steps, the computerprocessor generates an output indicating a relationship between theanticipated longitudinal insertion path of the tool and the designatedlocation. For some applications, the computer processor generates anoutput on the display, e.g., as shown in FIGS. 16A and 16B.Alternatively or additionally, the computer processor may generateinstructions to the operator to redirect the tool. Further alternativelyor additionally, the computer processor may generate an alert (e.g., anaudio or visual alert) in response to detecting that the tool isanticipated to enter a region that should be avoided (such as the spinalcord), or is anticipated to exit the vertebra sideways in the otherdirection.

Referring again to step 90 of FIG. 7, for some applications, computerprocessor 22 determines a location of a portion of the tool with respectto the vertebra, within the x-ray images, by means of image processing,as described hereinabove. Based upon the identified location of theportion of the tool within the x-ray images, and the registration of thefirst and second 2D x-ray images to the 3D image data, the computerprocessor determines the location of the portion of the tool withrespect to the 3D image data. For some applications, in responsethereto, the computer processor shows an image of the tool itself, or asymbolic representation thereof, overlaid upon the 3D image data.Alternatively or additionally, the computer processor derives arelationship between the location of the portion of the tool withrespect to the 3D image data and a given location within the 3D imagedata, and generates an output that is indicative of the relationship. Asdescribed hereinabove, the registration of the 2D images to the 3D imagedata is typically performed with respect to individual vertebrae.Therefore, even is the subject has moved between the acquisition of the3D image data and the acquisitions of the 2D images, the techniquesdescribed herein are typically effective.

For some applications, the computer processor generates an output thatis indicative of the distance of the tip of the tool from the spinalcord and/or outer vertebral border, e.g., using numbers or colorsdisplayed with respect to the 3D image data. For some applications, thecomputer processor outputs instructions (e.g., textual, graphical, oraudio instructions) indicating that the tool should be redirected. Forsome applications, as an input to this process, the computer processordetermines or receives a manual input indicative of a direction ororientation from which the tool is inserted (e.g., from top or bottom,or left or right).

Reference is now made to FIG. 17A shows an AP x-ray of two tools 176Land 176R being inserted into a vertebra through, respectively, 10-11o'clock and 1-2 o'clock insertion windows, and to FIG. 17B, which showsa corresponding lateral x-ray image to FIG. 17A, the images beingacquired in accordance with prior art techniques. As describedhereinabove, in many cases, during spinal surgery, two or more tools areinserted into a vertebra, for example, from the 10 o'clock to 11 o'clockinsertion window and from the 1 o'clock to 2 o'clock insertion window,with the process repeated, as applicable, for one or more furthervertebrae. Within the AP x-ray view, the tools inserted into respectivewindows are typically discernible from one another, as shown in FIG.17A. Furthermore, with reference to FIGS. 6A-B, for some applications,within the AP view, the computer processor discerns between toolinserted via the respective insertion windows based upon thearrangements of marker sets 50A and 50B (and in some cases 50C).However, if the tools are of identical or similar appearance, then fromsome imaging directions it is challenging to identify which tool iswhich. In particular, it is challenging to identify which tool is whichin lateral x-ray views, as may be observed in FIG. 17B. In general, itis possible to discern between tools in images acquired along thedirection of insertion, and more difficult to discern between tools inimages acquired along other directions.

For some applications, as a solution to the above-described challenge,the computer processor provides the operator with an interface toidentify one or more of the tools within an x-ray image, e.g., bymatching tools within a first image that was acquired from a view inwhich the tools are discernable from one another (e.g., the AP view) totools within a second image that was acquired from a view in which thetools are not discernable from one another (e.g., the lateral view).Alternatively or additionally, since the tools are typically insertedsequentially and all of the tool insertions are performed under x-rayimaging, when the first tool is inserted, images are acquired from botha first image view in which tools are discernable from one another(e.g., the AP view) and a second image view in which tools are notdiscernable from one another (e.g., the lateral view) are acquired. Thecomputer processor thereby identifies the tool as being the first tool,even in the image acquired from the second image view in which tools arenot discernable from one another. Subsequently, when the second tool isinserted and the images (from the same, or similar, two views) arereacquired, the computer processor is able to identify the second tool,even in images acquired from the second image view in which tools arenot discernable from one another, since the identification of the firsttool has already been performed. The computer processor then keeps trackof which tool is which along the sequence of x-ray images. For someapplications, once the computer processor has determined which tool iswhich, the computer processor indicates which tool is which at allrelevant stages throughout the procedure (e.g., by color-coding orlabelling the tools), with respect to all x-ray views, and/or withrespect to the 3D image data.

Referring again to step 90 of FIG. 7, for some applications, rather thandisplaying the tool, a representation thereof, and/or a path thereofupon a 3D image, the computer processor drives the display to displaythe tool, a representation thereof, and/or a path thereof upon a 2Dcross-section of the vertebra that is derived from the 3D image. Forsome applications, the computer processor determines the location of thecenterline of the tool shaft, by means of image processing. For example,the computer processor may use techniques for automatically identifyinga centerline of an object as described in US 2010-0161022 to Tolkowsky,which is incorporated herein by reference. For some applications, thecomputer processor drives the display to display the centerline of thetool upon the 3D image data, the end of the centerline indicating thelocation of the tool tip within the 3D image data. Alternatively oradditionally, the computer processor drives the display to display anextrapolation of the centerline of the tool upon the 3D image data, theextrapolation of the centerline indicating an anticipated path of thetool with respect to the 3D image data. For some applications, thecomputer processor drives the display to display a dot at the end of theextrapolated centerline upon the 3D image data, the dot representing theanticipated location of the tip of the tool.

For some applications, the computer processor drives the display todisplay in a semi-transparent format a 3D image of the vertebra with thetool, a representation thereof, and/or a path thereof disposed insidethe 3D image. Alternatively or additionally, the computer processordrives the display to rotate the 3D image of the vertebra automatically(e.g., to rotate the 3D image back-and-forth through approximately 30degrees). For some applications, the computer processor retrieves animage of a tool of the type that is being inserted from a library andoverlays the image upon the derived centerline upon the 3D image data.Typically, the tool is placed along the centerline at an appropriatescale with the dimensions being derived from the 3D image data. For someapplications, a cylindrical representation of the tool is overlaid uponthe derived centerline upon the 3D image data. For some applications,any one of the above representations is displayed relative to apredesignated tool path, as derived automatically by processor 22, or asinput manually by the surgeon during a planning stage.

Referring again to FIG. 2, tool insertion into a vertebra must avoid thespinal cord 42, and at the same time needs to avoid exiting the vertebrafrom the sides, leaving only two narrow tool insertion windows 44, oneither side of the vertebra. Typically, the greater the level ofprotrusion of a tool or implant into the spinal cord, the worse theclinical implications. For some applications, volumes within the 3Dimage of the vertebra (and/or a cross-sectional image derived therefrom)are color coded (e.g., highlighted) to indicate the level ofacceptability (or unacceptability) of protrusion into those volumes. Forsome applications, during the procedure, the computer processordetermines the location of the tool with respect to the 3D image data,and in response thereto, the computer processor drives the display tohighlight a vertebral volume into which there is a protrusion that isunacceptable. For some applications, the computer processor drives thedisplay to display a plurality (e.g., 2-6) of, typically concentric,cylinders within the 3D image of the vertebra, the cylinders indicatingrespective levels of acceptability of protrusion of a tool into thevolumes defined by the cylinders. During the procedure, the computerprocessor determines the location of the tool with respect to the 3Dimage data, and in response thereto, the computer processor drives thedisplay to highlight the cylinder in which the tool or a portion thereofis disposed, and/or is anticipated to enter. For some applications, thecomputer processor performs the above-described functionalities, but notwith respect to the tool that is currently being inserted (which may bea narrow tool, such as a needle), rather with respect to the eventualimplant (e.g., a pedicle screw, which typically has a larger diameter)that will be positioned later using the current tool. For someapplications, the computer processor performs the above-described stepswith respect to a 2D cross-sectional image that is derived from the 3Dimage data. For such cases, rectangles, rather than cylinders aretypically used to represent the respective levels of acceptability ofprotrusion of a tool into the areas defined by the rectangles.

For some applications, the processor allows a 3D image of the vertebrawith the tool, a representation of the tool, and/or a path of the toolindicated within the image to be rotated, or the processor rotates theimage automatically, in order for the user to better understand the 3Dplacement of the tool. It is noted that, since the images of thevertebra and the tool were input from different imaging sources, thesegmented data of what is the tool (or its representation) and what isthe vertebra is in-built (i.e., it is already known to the computerprocessor). For some applications, the computer processor utilizes thisin-built segmentation to allow the operator to virtually manipulate thetools with respect to the vertebra. For example, the operator mayvirtually advance the tool further along its insertion path, or retractthe tool and observe the motion of the tool with respect to thevertebra. For some applications, the computer processor automaticallyvirtually advances the tool further along its insertion path, orretracts the tool with respect to the vertebra in the 3D image data.

For some applications, accuracy of determining the position of theportion of the tool within the 3D image data is enhanced by registeringthree 2D x-ray images to the 3D image data, the images being acquiredfrom respective, different views from one another. Typically, for suchapplications, an oblique x-ray image view is used in addition to AP andlateral views. For some applications, accuracy of determining theposition of the portion of the tool within the 3D image data is enhancedby using x-ray images in which multiple portions of the tool, orportions of multiple tools, are visible and discernible from one anotherin the x-ray images. For some applications, the tools are discerned fromone another based on a manual input by the operator, or automatically bythe computer processor. For some applications, accuracy of determiningthe position of the portion of the tool within the 3D image data isenhanced by referencing the known shapes and/or dimensions of radiopaquemarkers 52 as described hereinabove.

Reference is now made to FIG. 18, which is a schematic illustration ofJamshidi™ needle 36 with a radiopaque clip 180 attached thereto, inaccordance with some applications of the present invention. For someapplications, accuracy of determining the position of the portion of thetool within the 3D image data is enhanced by adding an additionalradiopaque element to the tool (such as clip 180), such that the toolhas at least two identifiable features in each 2D image, namely, itsdistal tip and the additional radiopaque element. For some applications,the additional radiopaque element is configured to be have a defined 3Darrangement such that the additional radiopaque element providescomprehension of the orientation of the tool. For example, theadditional radiopaque element may include an array of radiopaquespheres. For some applications, the additional radiopaque elementfacilitates additional functionalities, e.g., as described hereinbelow.For some applications, the tool itself includes more than one radiopaquefeature that is identifiable in each 2D x-ray image. For suchapplications, an additional radiopaque element (such as clip 180) istypically not attached to the tool.

For some applications, the imaging functionalities described above withreference to the 3D image data are performed with respect to the 2Dx-ray images, based upon the coregistration of the 2D images to the 3Dimage data. For example, the tool may be color-coded in the x-ray imagesaccording to how well the tool is placed. For some applications, if thetool is placed incorrectly, the computer processor drives the display toshow how the tool should appear when properly placed, within the 2Dx-ray images.

Reference is now made to FIGS. 19A and 19B, which show examples of APx-ray images and corresponding lateral x-ray images of a vertebra, atrespective stages of the insertion of a tool into the vertebra, inaccordance with some applications of the present invention. Reference isalso made to step 88 of FIG. 7. A common practice in spinal surgery thatis performed under x-ray is to use two separate c-arm poses (typicallyany two of AP, lateral and oblique) to gain partial 3D comprehensionduring tool insertion and/or manipulation. This typically requiresmoving the C-arm back and forth, and exposes the patient to a highradiation dose.

For some applications of the present invention, images are initiallyacquired from two poses, which correspond to respective image views. Forexample, FIG. 19A shows examples of AP and lateral x-ray images of atool being inserted dorsally into a vertebra. Subsequently, the C-arm ismaintained at a single pose for repeat acquisitions during toolinsertion and/or manipulation, but the computer processor derives theposition of the tool with respect to the vertebra in additional x-rayimaging views, and drives the display to display the derived position ofthe tool with respect to the vertebra in the additional x-ray imageviews. For example, FIG. 19B shows an example of an AP image of the tooland the vertebra of FIG. 19A, but with the tool having advanced furtherinto the vertebra relative to FIG. 19A. Based upon the AP image in whichthe tool has advanced the computer processor derives the new, calculatedposition of the tool with respect to the lateral x-ray imaging view, anddrives the display to display a representation 190 of the new toolposition upon the lateral image. Typically, the new, calculated toolposition is displayed upon the lateral image, in addition to thepreviously-imaged position of the tool tip within the lateral image, asshown in FIG. 19B. Typically, the computer processor derives thelocation of portion of the tool with respect to one of the two original2D x-ray image views, based upon the current location of the portion ofthe tool as identified within a current 2D x-ray image, and arelationship that is determined between images that were acquired fromthe two original 2D x-ray image views, as described in further detailhereinbelow.

For some applications, the repeat acquisitions are performed from a 2Dx-ray image view that is the same as one of the original 2D x-ray imageviews, while for some applications the repeat acquisitions are performedfrom a 2D x-ray image view that is different from both of the original2D x-ray image views. For some applications, in the subsequent step, thetool within the vertebra is still imaged periodically from one or moreadditional 2D x-ray image views, in order to verify the accuracy of theposition of the tool within the additional views that was derived by thecomputer processor, and to correct the positioning of the tool withinthe additional 2D x-ray image views if necessary. For some applications,the C-arm is maintained at a single pose (e.g., AP) for repeatacquisitions during tool insertion and/or manipulation, and the computerprocessor automatically derives the location of portion of the tool withrespect to the 3D image data of the vertebra, and updates the image ofthe tool (or a representation thereof) within the 3D image data.

Typically, applications as described with reference to FIGS. 19A-B areused with a tool that is inserted into the skeletal anatomy along alongitudinal (i.e., a straight-line, or generally-straight-line)insertion path. For some applications, the techniques are used with atool that is not inserted into the skeletal anatomy along astraight-line insertion path. For such cases, the computer processortypically determines the non-straight line anticipated path of progressof the tool by analyzing prior progress of the tool, and/or by observinganatomical constraints along the tool insertion path and predictingtheir effect. For such applications, the algorithms describedhereinbelow are modified accordingly.

For some applications, the techniques described with reference to FIGS.19A-B are performed with respect to a primary x-ray imaging view whichis typically from the direction along which the intervention isperformed (and typically sets 50 of markers 52 are placed on or near thesubject such that the markers appear in this imaging view), and asecondary direction from which images are acquired to provide additional3D comprehension. In cases in which interventions are performeddorsally, the primary x-ray imaging view is typically generally AP,while the secondary view is typically generally lateral.

For some applications, computer processor 22 uses one of the followingalgorithms to perform the techniques described with reference to FIGS.19A-B.

Algorithm 1:

-   -   1. The original two 2D x-ray images X1 and X2 are registered to        3D image data using the techniques described hereinabove.    -   2. Based upon the registration, a generally-straight-line of the        tool TL (e.g., the centerline, or tool shaft), as derived from        the 2D x-ray images, is positioned with respect to the 3D image        data as TL-3D.    -   3. The generally-straight-line of the tool with respect to the        3D image data is extrapolated to generate a forward line F-TL3D        with respect to the 3D image data.    -   4. When the tool is advanced, a new 2D x-ray X1{circumflex over        ( )} is acquired from one of the prior poses only, e.g., from        the same pose from which the original X1 was acquired.        (Typically, to avoid moving the C-arm, this is the pose at which        the most recent of the two previous 2D x-rays was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 in the two images. For some applications, if            there has been movement, then Algorithm 2 described            hereinbelow is used.    -   5. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image        X1{circumflex over ( )}. This is denoted TPX1{circumflex over        ( )}.    -   6. The computer processor registers 2D x-ray image X1{circumflex        over ( )} to the 3D image data using the DRR that matches the        first x-ray view. It is noted that since pose did not change        between the acquisitions of X1 and X1{circumflex over ( )}, the        DRR that matches x-ray X1{circumflex over ( )} is same as for        x-ray X1. Therefore, there is no need to re-search for the best        DRR to match to x-ray X1{circumflex over ( )}.    -   7. The computer processor draws a line with respect to the 3D        image data from C1 through TPX1{circumflex over ( )}.    -   8. The computer processor identifies the intersection of that        line with the F-TL3D line as the new location of the tip, with        respect to the 3D image data. It is noted that in cases in which        the tool has been retracted, the computer processor identifies        the intersection of the line with the straight-line of the tool        with respect to the 3D image data TL-3D, rather than with        forward line F-TL3D with respect to the 3D image data.    -   9. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to the 3D image data, or with respect to x-ray image X2.

Algorithm 2:

-   -   1. The original two 2D x-ray images X1 and X2 are registered to        3D image data using the techniques described hereinabove.    -   2. Based upon the registration, a generally-straight-line TL of        the tool (e.g., the centerline, or tool shaft) as derived from        the x-ray images is positioned with respect to the 3D image data        as TL-3D.    -   3. The generally-straight-line of the tool with respect to the        3D image data is extrapolated to generate a forward line F-TL3D        with respect to the 3D image data.    -   4. When the tool is advanced, a new 2D x-ray X3 is acquired        from, typically, any pose, and not necessarily one of the prior        two poses.    -   5. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image X3.        This is denoted TPX3.    -   6. The computer processor registers 2D x-ray image X3 to the 3D        image data of the vertebra by finding a DRR that best matches 2D        x-ray image X3, using the techniques described hereinabove. The        new DRR has a corresponding simulated camera position C3.    -   7. The computer processor draws a line with respect to the 3D        image data from C3 through TPX3.    -   8. The computer processor identifies the intersection of that        line with the F-TL3D line as the new location of the tip, with        respect to the 3D image data. It is noted that in cases in which        the tool has been retracted, the computer processor identifies        the intersection of the line with the straight-line of the tool        with respect to the 3D image data TL-3D, rather than with        forward line F-TL3D with respect to the 3D image data.    -   9. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to the 3D image data, or with respect to x-ray image X1        and/or X2.

Algorithm 3:

Reference is now made to FIG. 20, which is a schematic illustration of athree-dimensional rigid jig 194 that comprises at least portions 196thereof that are radiopaque and function as radiopaque markers, theradiopaque markers being disposed in a predefined three-dimensionalarrangement, in accordance with some applications of the presentinvention. For some applications, as shown, radiopaque portions 196 areradiopaque spheres (which, for some applications, have different sizesto each other, as shown), and the spheres are coupled to one another byarms 198 that are typically radiolucent. Typically, the spheres arecoupled to one another via the arms, such that the spatial relationshipsbetween the spheres are known precisely.

The following algorithm is typically implemented by computer processor22 even in cases in which the x-ray images are not registered with 3Dimage data of the vertebra. Typically, this algorithm is for use with athree-dimensional radiopaque jig, such as jig 194, sufficient portionsof which are visible in all applicable x-ray images and can be used torelate them to one another. For some applications, the jig includes a 3Darray of radiopaque spheres, as shown in FIG. 20. For example, the jigmay be attached to the surgical table.

-   -   1. The original two 2D x-ray images X1 and X2 are registered to        one another, using markers of the jig as an anchor to provide a        3D reference frame.    -   2. When the tool is advanced, a new x-ray X1{circumflex over        ( )} is acquired from one of the prior poses, e.g., from the        same pose from which the original X1 was acquired. (Typically,        to avoid moving the C-arm, this is the pose at which the most        recent of the two-previous x-ray was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 (typically, relative to the subject's visible            skeletal portion), and/or portions 196 of jig 194            (typically, relative to the subject's visible skeletal            portion), in the two images. For some applications, if there            has been movement, then one of the other algorithms            described herein is used.    -   3. The computer processor identifies, by means of image        processing, the location of the tool's distal tip in image        X1{circumflex over ( )}. This is denoted TPX1{circumflex over        ( )}.    -   4. The computer processor registers 2D x-ray image X1{circumflex        over ( )} with X2 using the jig.    -   5. The computer processor calculates the new location of the        tool tip upon X2, based upon the registration.    -   6. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to x-ray image X2.

Algorithm 4:

The following algorithm is typically implemented by computer processor22 even in cases in which the x-ray images are not registered with 3Dimage data of the vertebra. Typically, this algorithm is for use with atool that has two or more identifiable points in each 2D x-ray image.For example, this algorithm may be used with a tool to which a clip, oranother radiopaque feature is attached as shown in FIG. 18.

-   -   1. Within the original two 2D x-ray images X1 and X2, the        computer processor identifies, by means of image processing, the        two identifiable points of the tool, e.g., the distal tip and        the clip.    -   2. The computer processor determines a relationship between X1        and X2, in terms of image pixels. For example:        -   a. In X1, the two-dimensional distances between the tool tip            and the clip are dx1 pixels horizontally and dy1 pixels            vertically.        -   b. In X2, the two-dimensional distances between the tool tip            and the clip are dx2 pixels horizontally and dy2 pixels            vertically        -   c. Thus, the computer processor determines a 2D relationship            between the two images based upon the ratios dx2:dx1 and            dy2:dy1.    -   3. When the tool is advanced, a new x-ray X1{circumflex over        ( )} is acquired from one of the prior poses, e.g., from the        same pose from which the original x-ray X1 was acquired.        (Typically, to avoid moving the C-arm, this will be the pose at        which the most recent of the previous x-rays was acquired.)        -   For some applications, the computer processor verifies that            there has been no motion of the C-arm with respect to the            subject, and/or vice versa, between the acquisitions of X1            and X1{circumflex over ( )}, by comparing the appearance of            markers 52 in the two images. For some applications, if            there has been movement, then one of the other algorithms            described herein is used.    -   4. The computer processor identifies, by means of image        processing, the tip of the tool in image X1{circumflex over        ( )}.    -   5. The computer processor determines how many pixels the tip has        moved between the acquisitions of images X1 and X1{circumflex        over ( )}.    -   6. Based upon the 2D relationship between images X1 and X2, and        the number of pixels the tip has moved between the acquisitions        of images X1 and X1{circumflex over ( )}, the computer processor        determines the new location of the tip of the tool in image X2.    -   7. The computer processor drives the display to display the tool        tip (or a representation thereof) at its new location with        respect to x-ray image X2.

With reference to FIGS. 19A and 19B, in general, the scope of thepresent invention includes acquiring 3D image data of a skeletalportion, and acquiring first and second 2D x-ray images, from respectivex-ray image views, of the skeletal portion and a portion of a toolconfigured to be advanced into the skeletal portion along a longitudinalinsertion path, while the portion of the tool is disposed at a firstlocation with respect to the insertion path. The location of a portionof the tool with respect to the skeletal portion is identified withinthe first and second 2D x-ray images, by computer processor 22 of system20, by means of image processing, and the computer processor registersthe 2D x-ray images to the 3D image data, e.g., using the techniquesdescribed herein. Thus, a first location of the portion of the tool withrespect to the 3D image data is determined. Subsequently, the tool isadvanced along the longitudinal insertion path with respect to theskeletal portion, such that the portion of the tool is disposed at asecond location along the longitudinal insertion path. Subsequent tomoving the portion of the tool to a second location along the insertionpath, one or more additional 2D x-ray images of at least the portion ofthe tool and the skeletal portion are acquired from a single image view.In accordance with respective applications, the single image view is thesame as one of the original 2D x-ray image views, or is a third,different 2D x-ray image view. Computer processor 22 of system 20identifies the second location of the portion of the tool within the oneor more additional 2D x-ray images, by means of image processing, andderives the second location of the portion of the tool with respect tothe 3D image data, based upon the second location of the portion of thetool within the one or more additional 2D x-ray images, and thedetermined first location of the portion of the tool with respect to the3D image data. Typically, an output is generated in response thereto(e.g., by displaying the derived location of the tool relative to thex-ray image view with respect to which the location has been derived).

In accordance with some applications, first and second 2D x-ray imagesare acquired, from respective x-ray image views, of the skeletal portionand a portion of a tool configured to be advanced into the skeletalportion along a longitudinal insertion path, while the portion of thetool is disposed at a first location with respect to the insertion path.The location of a portion of the tool with respect to the skeletalportion is identified within the first and second 2D x-ray images, bycomputer processor 22 of system 20, by means of image processing, andthe computer processor determines a relationship between the first andsecond 2D x-ray images, e.g., using any one of algorithms 1-4 describedhereinabove. Subsequently, the tool is advanced along the longitudinalinsertion path with respect to the skeletal portion, such that theportion of the tool is disposed at a second location along thelongitudinal insertion path. Subsequent to moving the portion of thetool to the second location along the insertion path, one or moreadditional 2D x-ray images of at least the portion of the tool and theskeletal portion are acquired from a single image view. In accordancewith respective applications, the single image view is the same as oneof the original 2D x-ray image views, or is a third, different 2D x-rayimage view. Computer processor 22 of system 20 identifies the secondlocation of the portion of the tool within the one or more additional 2Dx-ray images by means of image processing, and derives the secondlocation of the portion of the tool with respect to one of the original2D x-ray image views, based upon the second location of the portion ofthe tool that was identified within the additional 2D x-ray image, andthe determined relationship between the first and second 2D x-rayimages. Typically, an output is generated in response thereto (e.g., bydisplaying the derived location of the tool relative to the x-ray imageview with respect to which the location has been derived).

Some examples of the applications of the techniques described withreference to FIGS. 19A and 19B are as follows. For an intervention thatis performed dorsally, initially x-rays may be acquired from lateral andAP views. Subsequent x-ray may be generally acquired from an AP viewonly (with optional periodic checks from the lateral view, as describedhereinabove), with the updated locations of the tool with respect to thelateral view being derived and displayed. It is noted that although, inthis configuration, the C-arm may disturb the intervention, the AP viewprovides the best indication of the location of the tool relative to thespinal cord. Alternatively, subsequent x-ray may be generally acquiredfrom a lateral view only (with optional periodic checks from the AP viewas described hereinabove), with the updated locations of the tool withrespect to the AP view being derived and displayed. Typically, for suchapplications, sets 50 of markers 52 are placed on the patient such thatat least one set of markers is visible from the lateral view. Furtheralternatively, subsequent x-ray may be generally acquired from anoblique view only (with optional periodic checks from the lateral and/orAP view as described hereinabove), with the updated locations of thetool with respect to the AP and/or lateral view being derived anddisplayed. It is noted that the above applications are presented asexamples, and the scope of the present invention includes using thetechniques described with reference to FIGS. 19A and 19B withinterventions that are performed on any portion of the skeletal anatomy,from any direction of approach, and with any type of x-ray image views,mutatis mutandis.

For some applications, the image of the tool (a representation thereof,and/or a path thereof) as derived from the 2D images is overlaid uponthe 3D image data of the vertebra as a hologram. As noted hereinabove,since, in accordance with such applications, the images of the vertebraand the tool (or a representation thereof) are input from differentimaging sources, the segmented data of what is the tool (or itsrepresentation) and what is the vertebra is in-built (i.e., it isalready known to the computer processor). For some applications, thecomputer processor utilizes this in-built segmentation to allow theoperator to virtually manipulate the tool with respect to the vertebra,within the hologram. For example, the operator may virtually advance thetool further along its insertion path, or retract the tool and observethe motion of the tool with respect to the vertebra. Or, the computerprocessor may automatically drive the holographic display to virtuallyadvance the tool further along its insertion path, or retract the tool.For some applications, similar techniques are applied to other tools andbodily organs, mutatis mutandis. For example, such techniques could beapplied to a CT image of the heart in combination with 2D angiographicimages of a catheter within the heart.

For some applications, an optical camera is used to acquire opticalimages of a tool. For example, optical camera 114, which is disposed onx-ray C-arm 34, as shown in FIG. 1, may be used. Alternatively oradditionally, an optical camera may be disposed on a separate arm, maybe handheld, may be the back camera of a display such as a tablet ormini-tablet device, may be placed on the surgeon's head, may be placedon another portion of the surgeon's body, and/or may be held by anothermember of the surgical staff. Typically, the computer processor derivesthe location of the tool with respect to the 3D image data, based upon2D images in which the tool was observed and using the registrationtechniques described hereinabove. For some applications, in addition,the computer processor identifies the tool within an optical imageacquired by the optical camera. For some such applications, the computerprocessor then overlays the 3D image data upon the optical image byaligning the location of the tool within the 3D image data and thelocation of the tool within the optical image. The computer processorthen drives an augmented reality display to display the 3D image dataoverlaid upon the optical image. Such a technique may be performed usingany viewing direction of the optical camera within which the tool isvisible, and typically without having to track the position of thesubject with respect to the optical camera.

For some applications, the location of the tool within the optical imagespace is determined by using two or more optical cameras, and/or one ormore 3D optical cameras. For some applications, even with one 2D opticalcamera, the 3D image data is overlaid upon the optical image, byaligning two or more tools from each of the imaging modalities. For someapplications, even with one 2D optical camera and a single tool, the 3Dimage data is overlaid upon the optical image, by acquiring additionalinformation regarding the orientation (e.g., rotation) of the tool,and/or the depth of the tool below the skin. For some applications, suchinformation is derived from 3D image data from which the location of theskin surface relative to the vertebra is derived. Alternatively oradditionally, such information is derived from an x-ray image in whichthe tool and the subject's anatomy are visible. Alternatively oradditionally, such information is derived from the marker set as seen inan x-ray image in which the tool and the subject's anatomy are visible.

As noted hereinabove, since the images of the vertebra and the tool (ora representation thereof) are input from different imaging sources, thesegmented data of what is the tool (or its representation) and what isthe vertebra is in-built (i.e., it is already known to the computerprocessor). For some applications, the computer processor utilizes thisin-built segmentation to allow the operator to virtually manipulate thetool with respect to the vertebra, within an augmented reality display.For example, the operator may virtually advance the tool further alongits insertion path, or retract the tool and observe the motion of thetool with respect to the vertebra. Or, the computer processor mayautomatically drive the augmented reality display to virtually advancethe tool further along its insertion path, or retract the tool.

Although some applications of the present invention have been describedwith reference to 3D CT image data, the scope of the present inventionincludes applying the described techniques to 3D MRI image data. Forsuch applications, 2D projection images (which are geometricallyanalogous to DRRs that are generated from CT images) are typicallygenerated from the MRI image data and are matched to the 2D images,using the techniques described hereinabove. For some applications, othertechniques are used for registering MRI image data to 2D x-ray images.For example, pseudo-CT image data may be generated from the MRI imagedata (e.g., using techniques as described in “Registration of 2D x-rayimages to 3D MRI by generating pseudo-CT data” by van der Bom et al.,Physics in Medicine and Biology, Volume 56, Number 4), and the DRRs thatare generated from the pseudo-CT data may be matched to the x-rayimages, using the techniques described hereinabove.

For some applications, MRI imaging is used during spinal endoscopy, andthe techniques described herein (including any one of the stepsdescribed with respect to FIG. 7) are used to facilitate performance ofthe spinal endoscopy. Spinal endoscopy is an emerging procedure that isused, for example, in spinal decompression. By using an endoscope,typically, tools can be inserted and manipulated via a smaller incisionrelative to current comparable surgery that is used for similarpurposes, such that a smaller entry space provides a larger treatmentspace than in traditional procedures. Typically, such procedures areused for interventions on soft tissue, such as discs. Such tissue istypically visible in MRI images, but is less, or not at all, visible inCT images or in 2D x-ray images. Traditionally, such procedures commencewith needle insertion under C-Arm imaging. A series of dilators are theninserted to gradually broaden the approach path. Eventually, an outertube having a diameter of approximately 1 cm is then kept in place andan endoscope is inserted therethrough. From this point on, the procedureis performed under endoscopic vision.

For some applications, level verification as described hereinabove isapplied to a spinal endoscopy procedure in order to determine thelocation of the vertebra with respect to which the spinal endoscopy isto be performed. Alternatively or additionally, the entry point for thespinal endoscopy may be determined using bidirectional mapping ofoptical images and x-ray images, as described hereinabove. For someapplications, MRI image data are registered to intraprocedural 2D x-rayimages. Based upon the registration, additional steps which aregenerally as described hereinabove are performed. For example, theneedle, dilator, and/or endoscope (and/or a representation thereof,and/or a path thereof) may be displayed relative to a target within theMRI image data (e.g., a 3D MRI image, a 2D cross-section derived from 3DMRI image data, and/or a 2D projection image derived from 3D MRI imagedata). For some applications, endoscopic image data are coregistered tointraprocedural 2D x-ray images. For example, respective endoscopicimage data points may be coregistered with respective locations withinthe intraprocedural images. For some applications, the coregisteredendoscopic image data are displayed with the intraprocedural images,together with an indication of the coregistration of respectiveendoscopic image data points with respective locations within theintraprocedural images. Alternatively or additionally, endoscopic imagedata are coregistered to MRI image data. For example, respectiveendoscopic image data points may be coregistered with respectivelocations within the MRI image data. For some applications, thecoregistered endoscopic image data are displayed with the MRI imagedata, together with an indication of the coregistration of respectiveendoscopic image data points with respective locations within the MRIimage data.

For some applications, the techniques described herein are performed incombination with using a robotic arm, such as a relatively low-costrobotic arm having 5-6 degrees of freedom. In accordance with someapplications, the robotic arm is used for holding, manipulating, and/oractivating a tool, and/or for operating the tool along a pre-programmedpath. For some applications, computer processor 22 drives the roboticarm to perform any one of the aforementioned operations responsively toimaging data, as described hereinabove.

Reference is now made to FIG. 21A, which shows examples of x-ray imagesof an image calibration jig generated by a C-arm that uses an imageintensifier (on the left), and by a C-arm that uses a flat-paneldetector (on the right), such images reflecting prior art techniques.Reference is also made to FIG. 21B, which shows an example of an x-rayimage acquired by a C-arm that uses an image intensifier, the imageincluding a radiopaque component 200 that corresponds to a portion of atool that is known to be straight, and a dotted line 210 overlaid uponthe image indicating how a line (for example, a centerline) defined bythe straight portion would appear if distortions in the image arecorrected, in accordance with some applications of the presentinvention.

As may be observed in the example shown in FIG. 21A, in x-ray imagesgenerated by a C-Arm that uses an image intensifier, there is typicallyimage distortion, which increases toward the periphery of the image. Bycontrast, in images generated using a flat-panel detector, there istypically no distortion. For some applications of the present invention,distortions in x-ray images generated by a C-Arm that uses an imageintensifier are at least partially corrected automatically, by means ofimage processing. For example, the distortion of such images may becorrected in order to then register the corrected image to a 3D imagedata, using the techniques described hereinabove.

Referring to FIG. 21B, for some applications such an x-ray image is atleast partially corrected by computer processor 22 identifying, by meansof image processing, a radiopaque component 200 of an instrument withina portion of the radiographic image. For some applications, theradiopaque component is a portion of the tool that is known to bestraight, a component having a different known shape, and/or two or morefeatures that are disposed in known arrangement with respect to oneanother. For example, the straight shaft of a Jamshidi™ needle may beidentified.

For some applications, in order to at least partially correct an x-rayimage comprising a radiopaque component that is known to be straight,the computer processor uses techniques for automatically identifying acenterline of an object, for example, as described in US 2010-0161022 toTolkowsky, which is incorporated herein by reference, to generate acenterline of said component. Typically, the computer processor then atleast partially corrects the image distortion, in at least a portion ofthe image in which the component that is known to be straight isdisposed, by deforming the portion of the radiographic image, such thatthe centerline of the radiopaque component of the instrument that isknown to be straight appears straight within the radiographic image.FIG. 19B shows an example of how an x-ray image, prior to correcting itsdistortion, comprises component 200 that is known to be straight and yetdoes not appear straight within the image, as can be observed relativeto straight line 210. For some applications, two or more such componentsare identified in respective portions of the image, and distortion ofthose portions of the image are corrected accordingly. For someapplications, distortions in portions of the image in which no suchcomponents are disposed are corrected, based upon distortion correctionparameters that are determined by means of the radiopaque component thatis known to be straight, or known to have a different known shape.

For some applications of the present invention, techniques describedhereinabove are combined with a system that determines the location ofthe tip of a tool with respect to a portion of the subject's body by (a)calculating a location of a proximal portion of the tool that isdisposed outside the subject's body, and (b) based upon the calculatedposition of the proximal portion of the tool, deriving a location of atip of the tool with respect to the portion of the subject's body withrespect to the 3D image data. For example, such techniques may be usedwith a navigation system that, for example, may include the use of oneor more location sensors that are attached to a portion of a tool thatis typically disposed outside the subject's body even during theprocedure. (It is noted that the location sensors that are disposed uponthe tool may be sensors that are tracked by a tracker that is disposedelsewhere, or they may be a tracker that tracks sensors that aredisposed elsewhere, and thereby acts a location sensor of the tool.) Forexample, a tool may be inserted into the subject's vertebra, such thatits distal tip (or a distal portion of the tool) is disposed inside thevertebra, and a location sensor may be disposed on a proximal portion ofthe tool that is disposed outside the subject's body. The navigationsystem typically derives the location of the tip of the tool (or adistal portion of the tool), by detecting the location(s) of thelocation sensor(s) that are disposed on the proximal portion of thetool, and then deriving the location of the tip of the tool (or a distalportion of the tool) based upon an assumed location of the distal tip ofthe tool (or a distal portion of the tool) relative to the locationsensor(s). The navigation system then overlays the derived location ofthe tip of the tip of the tool (or a distal portion of the tool) withrespect to the vertebra upon previously acquired 3D image data (e.g.,images acquired prior to the subject being placed in the operating room,or when the subject was in the operating room, but typically prior tothe commencement of the intervention). Alternatively or additionally,the location of a proximal portion of the tool that is disposed outsidethe subject's body may be calculated by video tracking the proximalportion of the tool, and/or by means of tracking motion of a portion ofa robot to which the proximal portion of the tool is coupled, relativeto a prior known position, e.g., based upon the values of the joints ofthe robot relative to the corresponding values of the joints of therobot at the prior known position.

In such cases, there may be errors associated with determining thelocation of the tip of the tool (or a distal portion of the tool), basedupon the assumed location of the distal tip of the tool (or a distalportion of the tool) relative to the location sensor(s) being erroneous,e.g., due to slight bending of the tool upon being inserted into thevertebra. Therefore, for some applications, during the procedure,typically periodically, 2D x-ray images are acquired within which theactual tip of tool (or distal portion of the tool) within the vertebrais visible. The location of the tip of the tool (or distal portion ofthe tool) with respect to the vertebra as observed in the 2D x-rayimages is determined with respect to the 3D image data, by registeringthe 2D x-ray images to the 3D image data. For example, the 2D x-rayimages may be registered to the 3D image data using techniques describedhereinabove. In this manner, the actual location of the tip of the tool(or distal portion of the tool) with respect to the vertebra isdetermined with respect to the 3D image data. For some applications, inresponse thereto, errors in the determination of the location of the tipof the tool (or distal portion of the tool) with respect to the vertebrawithin the 3D image space resulting from the navigation system, areperiodically corrected by system 20. For example, based upon thedetermined location of at least the tip of the tool (or distal portionof the tool), the computer processor may drive the display to update theindication of the location of the tip of the tool (or distal portion ofthe tool) with respect to the vertebra with respect to the 3D imagedata. For some applications, the navigation systems comprise the use ofaugmented reality, or virtual reality, or robotic manipulation of tools,or any combination thereof.

By way of illustration and not limitation, it is noted that the scope ofthe present invention includes applying the apparatus and methodsdescribed herein to any one of the following applications:

-   -   Multiple tool insertions (e.g., towards both pedicles) in the        same vertebra.    -   Any type of medical tool or implant, including, Jamshidi™        needles, k-wires, pedicle markers, screws, endoscopes, RF        probes, laser probes, injection needles, etc.    -   An intervention that is performed from a lateral approach, in        which case the functional roles of the AP and lateral x-ray        views described hereinabove are typically switched with one        another.    -   Interventions using x-ray views other than lateral and AP views        as an alternative or in addition to such views. For example,        oblique imaging views may be used.    -   An intervention that is performed from an anterior, oblique        and/or posterior interventional approach.    -   Interventions performed upon multiple vertebrae. Even for such        cases, the intra-operative x-ray images of the vertebrae are        typically registered with the 3D image data of the corresponding        vertebrae on an individual basis.    -   Interventions performed on discs in between vertebrae.    -   Interventions performed on nerves.    -   Tool insertion under x-ray in a video imaging mode.    -   Use of certain features of system 20 utilizing intraprocedural        2D x-ray imaging, but without utilizing preprocedural 3D        imaging.    -   Use of certain features of system 20 without some or all of the        above-described disposable items, such as drape 53.    -   Various orthopedic surgeries, such as surgeries performed on        limbs and/or joints.    -   Interventions in other body organs.

For some applications system 20 includes additional functionalities tothose described hereinabove. For example, the computer processor maygenerate an output that is indicative of a current level of accuracy(e.g., of verification of the vertebral level, determination of theinsertion site, and/or registration of the 3D image data to the 2Dimages), e.g., based upon a statistical calculation of the possibleerror. For some applications, the computer processor generates a promptindicating that a new x-ray from one or more views should be acquired.For example, the computer processor may generate such a prompt based onthe time elapsed since a previous x-ray acquisition from a given view,and/or based on the distance a tool has moved since a previous x-rayacquisition from a given view, and/or based on observed changes in theposition of markers 52 relative to the C-arm.

Applications of the invention described herein can take the form of acomputer program product accessible from a computer-usable orcomputer-readable medium (e.g., a non-transitory computer-readablemedium) providing program code for use by or in connection with acomputer or any instruction execution system, such as computer processor22. For the purpose of this description, a computer-usable or computerreadable medium can be any apparatus that can comprise, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Typically, the computer-usable or computer readablemedium is a non-transitory computer-usable or computer readable medium.

Examples of a computer-readable medium include a semiconductor or solidstate memory, magnetic tape, a removable computer diskette, arandom-access memory (RAM), a read-only memory (ROM), a rigid magneticdisk and an optical disk. Current examples of optical disks includecompact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W)and DVD. For some applications, cloud storage, and/or storage in aremote server is used.

A data processing system suitable for storing and/or executing programcode will include at least one processor (e.g., computer processor 22)coupled directly or indirectly to memory elements (such as memory 24)through a system bus. The memory elements can include local memoryemployed during actual execution of the program code, bulk storage, andcache memories which provide temporary storage of at least some programcode in order to reduce the number of times code must be retrieved frombulk storage during execution. The system can read the inventiveinstructions on the program storage devices and follow theseinstructions to execute the methodology of the embodiments of theinvention.

Network adapters may be coupled to the processor to enable the processorto become coupled to other processors or remote printers or storagedevices through intervening private or public networks. Modems, cablemodem and Ethernet cards are just a few of the currently available typesof network adapters.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the C programming language or similar programminglanguages.

It will be understood that blocks of the flowchart shown in FIGS. 7,14A, and 14B, combinations of blocks in the flowcharts, as well as anyone of the algorithms described herein, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general-purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer (e.g., computer processor 22) or other programmable dataprocessing apparatus, create means for implementing the functions/actsspecified in the flowcharts and/or algorithms described in the presentapplication. These computer program instructions may also be stored in acomputer-readable medium (e.g., a non-transitory computer-readablemedium) that can direct a computer or other programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable medium produce an article of manufactureincluding instruction means which implement the function/act specifiedin the flowchart blocks and algorithms. The computer programinstructions may also be loaded onto a computer or other programmabledata processing apparatus to cause a series of operational steps to beperformed on the computer or other programmable apparatus to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowcharts and/oralgorithms described in the present application.

Computer processor 22 and the other computer processors described hereinare typically hardware devices programmed with computer programinstructions to produce a special purpose computer. For example, whenprogrammed to perform the algorithms described herein, the computerprocessor typically acts as a special purpose skeletal-surgery-assistingcomputer processor. Typically, the operations described herein that areperformed by computer processors transform the physical state of amemory, which is a real physical article, to have a different magneticpolarity, electrical charge, or the like depending on the technology ofthe memory that is used.

It will be appreciated by persons skilled in the art that the presentinvention is not limited to what has been particularly shown anddescribed hereinabove. Rather, the scope of the present inventionincludes both combinations and subcombinations of the various featuresdescribed hereinabove, as well as variations and modifications thereofthat are not in the prior art, which would occur to persons skilled inthe art upon reading the foregoing description.

1-134. (canceled)
 135. Apparatus for use during a medical interventionin which a tool is used with respect to a portion of a body of asubject, and for use with: (a) a 3D imaging device configured to acquire3D image data of the portion of the subject's body, and (b) a 2D imagingdevice that is unregistered with respect to the subject's body andconfigured to sequentially: acquire a first 2D image of a distal portionof the tool and the portion of the subject's body from a first view,while the 2D imaging device is disposed at a first pose with respect tothe subject's body, be moved to a second pose with respect to thesubject's body, and while the 2D imaging device is at the second pose,acquire a second 2D image of the distal portion of the tool and theportion of the subject's body from a second view, the apparatuscomprising: at least one computer processor configured to: calculate alocation of the proximal portion of the tool that is disposed outsidethe subject's body, based upon the calculated location of the proximalportion of the tool, derive a location of the distal portion of the toolwith respect to the portion of the subject's body, with respect to the3D image data, subsequently: receive from the 2D imaging device thefirst and second 2D images of the distal portion of the tool and theportion of the subject's body, register the first and second 2D imagesof the portion of the subject's body to the 3D image data of the portionof the subject's body, identify a location of at least the distalportion of the tool within the first and second 2D images, by means ofimage processing, based upon the registration of the first and second 2Dimages of the portion of the subject's body to the 3D image data of theportion of the subject's body, and the identified location of at leastthe distal portion of the tool within the first and second 2D images,determine the location of at least the distal portion of the tool withrespect to the 3D image data of the portion of the subject's body, andbased upon the determined location of at least the distal portion of thetool, update the derived location of the distal portion of the tool withrespect to the portion of the subject's body, with respect to the 3Dimage data.
 136. The apparatus according to claim 135, wherein the atleast one computer processor is configured to register the-first andsecond 2D images of the portion of the subject's body to the 3D imagedata of the portion of the subject's body by: generating a plurality of2D projections from the 3D image data, and identifying respective firstand second 2D projections that match the first and second 2D images ofthe portion of the subject's body.
 137. The apparatus according to claim135, the apparatus further including one or more location sensorscoupled to the proximal portion of the tool, wherein the at least onecomputer processor is configured to calculate the location of theproximal portion of the tool that is disposed outside the subject's bodyby means of the one or more location sensors that are coupled to theproximal portion of the tool.
 138. The apparatus according to claim 135,wherein the at least one computer processor is configured to calculatethe location of the proximal portion of the tool that is disposedoutside the subject's body by video tracking the proximal portion of thetool.
 139. The apparatus according to claim 135, the apparatus furthercomprising a robot, the proximal portion of the tool being coupled to aportion of the robot, wherein the at least one computer processor isconfigured to calculate the location of the proximal portion of the toolthat is disposed outside the subject's body by calculating a location ofthe portion of the robot.
 140. (canceled)
 141. The apparatus accordingto claim 139, wherein the computer processor is configured to calculatethe location of the portion of the robot based on values of a pluralityof joints of the robot.
 142. The apparatus according to claim 139,wherein the computer processor is further configured to drive the robotto manipulate, place, or activate the tool in response to the derivedlocation of the distal portion of the tool with respect to the portionof the subject's body, with respect to the 3D image data.
 143. Theapparatus according to claim 139, wherein the computer processor isfurther configured to drive the robot to move the tool along a path inresponse to the derived location of the distal portion of the tool withrespect to the portion of the subject's body, with respect to the 3Dimage data.
 144. The apparatus according to claim 143, wherein operatingthe tool along a path comprises operating the tool along apre-programmed path.
 145. The apparatus according to claim 139, whereinthe computer processor is further configured to drive the robot tomanipulate, place, or activate the tool in response to the determinedlocation of at least the distal portion of the tool with respect to the3D image data of the portion of the subject's body.
 146. The apparatusaccording to claim 139, wherein the computer processor is furtherconfigured to drive the robot to move the tool along a path in responseto the determined location of at least the distal portion of the toolwith respect to the 3D image data of the portion of the subject's body.147. The apparatus according to claim 146, wherein operating the toolalong a path comprises operating the tool along a pre-programmed path.148. The apparatus according to claim 135, wherein the tool comprises animplant or an implant delivery probe.
 149. The apparatus according toclaim 135, wherein the apparatus is for use with a display, and whereinthe computer processor is further configured to: based upon thedetermined location of at least the distal portion of the tool, drivethe display to display the determined location of at least the distalportion of the tool with respect to the 3D image data of the portion ofthe subject's body.
 150. The apparatus according to claim 149, wherein:the computer processor is configured to, based upon the derived locationof the distal portion of the tool, drive the display to display anindication of the location of the distal portion of the tool withrespect to the portion of the subject's body, with respect to the 3Dimage data, and wherein driving the display to display the determinedlocation of at least the distal portion of the tool comprises drivingthe display to update the indication of the location of the distalportion of the tool with respect to the portion of the subject's body,with respect to the 3D image data.
 151. The apparatus according to claim135, wherein: the apparatus is for use with an augmented realitydisplay, and the computer processor is configured to: based upon thederived location of the distal portion of the tool, drive the augmentedreality display to display an indication of the location of the distalportion of the tool with respect to the portion of the subject's body,with respect to the 3D image data, and based upon the determinedlocation of at least the distal portion of the tool, drive the augmentedreality display to update the indication of the location of the distalportion of the tool on the augmented reality display.
 152. The apparatusaccording to claim 151, wherein the computer processor is furtherconfigured to allow an operator to virtually manipulate the tool withrespect to the portion of the subject's body, on the augmented realitydisplay.
 153. The apparatus according to claim 135, wherein: theapparatus is for use with a virtual reality display, and the computerprocessor is configured to: based upon the derived location of thedistal portion of the tool, drive the virtual reality display to displayan indication of the location of the distal portion of the tool withrespect to the portion of the subject's body, with respect to the 3Dimage data, and based upon the determined location of at least thedistal portion of the tool, drive the virtual reality display to updatethe indication of the location of the distal portion of the tool on thevirtual reality display.
 154. The apparatus according to claim 153,wherein the computer processor is further configured to allow anoperator to virtually manipulate the tool with respect to the portion ofthe subject's body, on the virtual reality display.
 155. The apparatusaccording to claim 135, wherein the apparatus further comprises a robot,the tool being coupled to a portion of the robot, and the computerprocessor is configured to calculate a location of the tool that isdisposed outside the subject's body by starting with the tool positionedat a known starting point relative to the subject's body andsubsequently recording motion of the tool from the known starting point.